« The Lean Architect » - Do we need Abstraction and Planning « on the Gemba » ?

<div dir="ltr" style="text-align: left;" trbidi="on"><br /><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US" style="line-height: 115%;">Today’s post is a first attempt at defining the role of a software architect in a “</span><span style="line-height: 115%;"><a href="http://informationsystemsbiology.blogspot.fr/2012/07/lean-software-factory.html"><span lang="EN-US">Lean Software Factory</span></a></span><span lang="EN-US" style="line-height: 115%;">”, that is, an agile development organization built around </span><span style="line-height: 115%;"><a href="http://www.lean-it-summit.com/"><span lang="EN-US">lean software</span></a></span><span lang="EN-US" style="line-height: 115%;"> principles. There is a natural tension between the two words “<b>lean</b>” and “<b>architect</b>”, so one may think that this title is an oxymoron: “lean management”, in general, tries to stay away from abstraction and too much planning, and promotes practical, concrete team-oriented problem solving “<i>on the gemba</i>”, where the action takes place. Agile development says that conception (requirements and design) evolves continuously while a software product is being developed. This is in contradiction with the architect’s role as defined in the <a href="http://en.wikipedia.org/wiki/V-Model_(software_development)">V software development model</a>, which is the product of Taylorism applied to development, where the Architect (capital A) thinks a grand design that is built by others. The team-oriented approach of lean software development relies on democracy and consensus, where each voice is heard, and where no “design architecture” can be accepted as a “non-negotiable” requirement. Similarly, a key practice of lean/agile software development is synchronicity (everyone working on the same “takt time”), which is implemented through stand-up meetings, <a href="http://www.shmula.com/obeya-communication-breakdown/385/">common project rooms</a>and intense real-time discussion with the product owner, which means that the very concept of “planning” and “design” seems in jeopardy. <o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US" style="line-height: 115%;"><br /></span></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US" style="line-height: 115%;"><span style="font-family: Verdana, sans-serif;">The first question I’d like to ask is, therefore, “<i>do we need an architectural process when software development is led in a lean/agile mode ?</i>”. My answer is a resounding “YES”, with the following three arguments:<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"></div><ul><li><span lang="EN-US" style="font-family: Verdana, sans-serif; line-height: 115%; text-align: left; text-indent: -18pt;">Agile or lean software development does not prevent from building software products that are actually complex and require a carefully-crafted form of internal organization. Let us recall that architecture is foremost about communication: its role is to provide organizations and actions with a narrative sense.  In a lean software development process, architecture is a “co-creation”:  architecture and design continuously evolve together with product development; they are not “completed” until the product is.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; line-height: 115%; text-align: left; text-indent: -18pt;">In a continuous development and integration process, we need an architectural framework to prevent from diverging (too many additions that move the product in different and opposite directions) and maintain a continuous and efficient <a href="http://en.wikipedia.org/wiki/Code_refactoring">refactoring</a>. Software code is a <i>living </i>object, which evolves together with the rythms of <a href="http://en.wikipedia.org/wiki/Scrum_(development)#Sprint">sprints</a> and update <a href="http://en.wikipedia.org/wiki/Continuous_delivery">deliveries</a>. As a living object, constant “<a href="http://www.craigberntson.com/blog/?p=964">gardening</a>” is required, which is guided by a vision (of what the garden should be), which is precisely what an “architecture” should be. As Jean-Claude Ameisen explains in his wonderful book “<a href="http://lasculptureduvivant.free.fr/">La Sculpture du Vivant</a>”, the birth of a living organism requires as much pruning and removal as creating and addition; this applies to software : there should be as much thoughts and care put into code pruning and removal as there is in lines-of-code addition.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; line-height: 115%; text-align: left; text-indent: -18pt;">A key principle from Toyota/Lean Management is the importance of a systemic vision that is shared by all actors (sharing a systemic understanding is a key role of the <a href="http://www.sixthsigma.com/2006/07/toyota-kaizen-obeya-room.htm">Obeya room</a>). Architecture in a lean software development process is not restricted to a few hand-picked specialists; <a href="http://carlokruger.com/?p=10">it is everyone’s responsibility</a>, especially developers’. A major feedback from SCRUM practitioners is the importance of “<a href="http://scribeandscrum.com/tag/design-review-committee/">design reviews</a>”. Organizing successful design reviews is better conducted by “experts”, who are people with experience (architecture is an art, much more than a science – one learns from previous mistakes). These are precisely “architects”, people who are well qualified to lead a successful review and to encourage learning/capitalization. <i>Agile </i>does not mean than everything needs to be rediscovered “on the gemba”; 30 years of software architecture have produced <a href="http://informationsystemsbiology.blogspot.fr/2009/04/selected-bibliography.html">both useful principles and tools</a>.</span></li></ul><br /> <div class="MsoNormal" style="text-align: justify;"><span lang="EN-US" style="line-height: 115%;"><span style="font-family: Verdana, sans-serif;">The second question to ask is then « <i>What role should an architect play in this emergent software development process ?</i> ». Part of the answer is common to the previous question and has been brushed in the previous three bullet points. The architect is required to shepherd the continuous refactoring and to lead the regular <i>design reviews</i>. This is quite different from the classical role as defined by the V-model. Let us emphasize the practical and cultural differences through three critical insights:<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"></div><ul><li><span lang="EN-US" style="font-family: Verdana, sans-serif; line-height: 115%; text-align: left; text-indent: -18pt;">An architect needs to be a <a href="http://storytellersforgood.com/2010/08/a-new-age-a-whole-new-mind/">story teller</a>. Agile processes suggest working on requirements expressed as “<a href="http://agiletools.wordpress.com/2007/11/24/task-boards-telling-a-compelling-agile-story/">story boards</a>”. Story boards make for “units of requirement” which are gathered into “backlogs”. The “architectural vision” also needs to be expressed as a set of “stories”.  Stories are a great form for expressing a requirement because they both carry meaning and they are robust. The heart of the agile process is that volatility, randomness, complexity of environment will force the product developers to adapt continuously during its completion. Strict and formal requirements tend to become obsolete or incompatible with one another too soon! Stories are robust because they can evolve; the combination of their meaning is context-dependent. I would qualify stories as “</span><span style="font-family: Verdana, sans-serif; line-height: 115%; text-align: left; text-indent: -18pt;"><a href="http://www.guardian.co.uk/books/2012/dec/16/antifragile-nassim-nicholas-taleb-review"><span lang="EN-US">antifragile</span></a></span><span lang="EN-US" style="font-family: Verdana, sans-serif; line-height: 115%; text-align: left; text-indent: -18pt;">” following Nassim Taleb’s insight: the volatility and variations that will necessarily occur will “shake” the story and often produce even more value than the original thought contained. Serendipity is produced by the random shocks between stories from the backlog and continuous feedback from the users.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; line-height: 115%; text-align: left; text-indent: -18pt;">An architect works in a “<i>pull mode</i>”, she is pulled by the need for advice, assistance or explanation from developers. This is very different of the “<i>push mode</i>” from the Taylor-style organizations where the architect would hand out a document to the developers: “<i>please implement the following architecture requirement R1 to R112</i>”. This requires some form of <a href="http://www.mindtools.com/pages/article/newPPM_82.htm">change management</a>, but this new way of working is quite fulfilling for an architect. She may feel the pleasure of participating to the actual creation, instead of being frustrated to see her “great principles, well-written in a forward-looking design document” ignored. This is the instantiation of a key lean principle that says that each actor in a process is at the service of the next actor (in the value creation process, from a chronological viewpoint). <a href="http://en.wikipedia.org/wiki/Kanban_(development)">Kanban</a>, both its industrial and <a href="http://en.wikipedia.org/wiki/Kanban_(development)">software version</a>, is the tool that puts this principle into practice.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; line-height: 115%; text-align: left; text-indent: -18pt;">Architects are stakeholders in the <a href="http://www.mountaingoatsoftware.com/scrum/product-backlog">SCRUM backlog</a>management: they ensure that performance tuning and bug fixing (especially “non-functional ones”) are maintained on the top of the stack.  Obviously the product owner is the key stakeholder, and these concerns should be hers/his as well. An architect is not a “<a href="http://www.lean.org/shook/displayobject.cfm?o=906">chief engineer</a>” in the Toyota sense (it is unclear to me if there is such a role in a lean software development process, although it is often used in the <a href="http://en.wikipedia.org/wiki/DevOps">Devops</a> context). As with any complex project, there exists a recursive/fractal structure associated with a product (systems and sub-systems). There are a number of “product owners” (for subsystems) and a master “product owner” who plays a role similar to that of a chief engineer. The architect is a member of the engineering team, a voice that reflects the value of long-term thinking, simplicity and “well-oiled” running. Using lean/agile does not remove the age-old conflict between short-term and long-term goals. Products tend to last (at least, customers wish they would), hence long-term characteristics are essential to a well-designed and well-built product.</span></li></ul><br /> <div class="MsoNormal" style="text-align: justify;"><span lang="EN-US" style="line-height: 115%;"><span style="font-family: Verdana, sans-serif;">To conclude this post, I would like to stand back and ask a more general question « <i>What is the role of the architect from a systemic viewpoint ?</i> ».<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"></div><ul><li><span lang="EN-US" style="font-family: Verdana, sans-serif; line-height: 115%; text-align: left; text-indent: -18pt;">At the side of the product owner, the lean architect carries the critical mission to build the “situation potential” in <a href="http://scienceofstrategy.org/main/content/article-pdf-41-future-potential">Sun Tzu’s</a> sense. The concept of “<a href="http://books.google.fr/books?id=OS_YqZhWuAsC&pg=PA39&lpg=PA39&dq=%22situation+potential%22+Jullien&source=bl&ots=hxZZQxGfaU&sig=hF7CzO0FIExCrhH-lEAPlILwWGo&hl=en&sa=X&ei=X0bpULfFLqWV0QX8mIDAAw&ved=0CDAQ6AEwAA#v=onepage&q=%22situation%20potential%22%20Julli">situation potential</a>” is the cornerstone of François Jullien’s book on efficiency. In a complex world, both <a href="http://organisationarchitecture.blogspot.fr/2009/05/simulation-jeux-et-previsions-face.html">François Jullien and Nassim Taleb</a> tell us that forecasting (as in “strategic planning”) does not work well any longer and one should become “opportunistic”, in the (positive) Chinese sense (i.e., be prepared to jump on opportunities). Being able to leverage the future, whatever it holds, is the heart of this “Art of War” strategy. It relies on agility: to react quickly as soon as the opportunity arises. It also relies on building the right set of capabilities: the situation potential. A “good architecture” is precisely a “situation potential”. This becomes even more obvious as soon as the necessity of a <b>platform strategy</b> occurs – the core message of “<a href="http://sivers.org/book/WhatWouldGoogleDo">What would Google do ?</a>” by Jeff Jarvis. Most successful software products are platforms (cf. the <a href="http://news.ycombinator.com/item?id=3102800">famous post by Steve Yegge</a>). Designing a platform requires every developer to be eager to contribute to an “open API culture”, to be ready  “to eat your own dogfood” (the best way to implement the feedback circle that is necessary to reach quality). However, designing a powerful platform is precisely an architectural task.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; line-height: 115%; text-align: left; text-indent: -18pt;">A key tenet of “<a href="http://informationsystemsbiology.blogspot.fr/2011/11/lean-it-devops-and-cloud-programming.html">lean software</a>”, as explained by Mary & Tom <a href="http://www.poppendieck.com/">Poppendieck</a>, is to free up creativity and innovation through cooperation. Architecture may be seen as the “<b>grammar</b>” for this cooperation. This is the core of “service-oriented architecture” (<a href="http://informationsystemsbiology.blogspot.fr/2009/05/soa-tale-of-two-cities.html">SOA</a>). A software architect, irrespectively of the software development method (agile or not), aims to develop <i>sharing </i>and <i>reuse </i>of software components. This brings us back to the critical role of architecture to develop a software platform. This is <a href="http://theleanstartup.com/">lean</a> & <a href="http://www.pretotyping.org/">pretotyping</a> at work: designing a platform is not (only) about thinking right, it is mostly about feedback loops and listening to users/customers. The value of a grammar is judged from the beautiful sentences that other write through it.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; line-height: 115%; text-align: left; text-indent: -18pt;">The more evolution we see in our current world, the more we need solid foundations to avoid losing our time building, destroying and re-building over and over. <a href="http://en.wikipedia.org/wiki/Multilayered_architecture">Layered architecture</a> stays a relevant paradigm in the world of “lean software” because layer structure tells us the order in which things should be built and re-built. Experience shows that one cannot rely only of “auto-organization”: too many iterations – that entail massive rework – are necessary until a fixed-point is reached. Also, the practice of 5S (</span><i style="font-family: Verdana, sans-serif; text-align: left; text-indent: -18pt;"><span lang="EN-US" style="background-color: white; line-height: 115%;">sorting, set in order, systematic cleaning, standardizing, and sustaining)</span></i><span lang="EN-US" style="background-color: white; font-family: Verdana, sans-serif; line-height: 115%; text-align: left; text-indent: -18pt;">, applied to software (as explained by Mary Poppendieck for instance)</span><span lang="EN-US" style="font-family: Verdana, sans-serif; line-height: 115%; text-align: left; text-indent: -18pt;">, is both relevant to code development and software architecture. There are many other interesting architectural contribution to a lean software product, such as the <a href="https://sites.google.com/a/gertrudandcope.com/www/thedciarchitecture">DCI architecture</a>, which may be found in the book “<a href="http://www.leansoftwarearchitecture.com/">Lean Software Architecture</a>”.</span></li></ul><br /> <div class="MsoNormal"><br /></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US" style="line-height: 115%;"><span style="font-family: Verdana, sans-serif;">This short post does not do justice to the complex topic of “lean software development” mixed with “software architecture”. The title, “Lean Architect”, applies to a long-term goal, which is to reconcile large-scale-systems-engineering with agile development methods. My own conviction is that “lean” – in the “<a href="http://en.wikipedia.org/wiki/The_Toyota_Way">Toyota Way</a>” sense -  is the appropriate framework for such an integration. Lean adds to agile methods the <a href="http://informationsystemsbiology.blogspot.fr/2012/12/lean-scrum-agile-and-extreme-programming.html">combination of a global/systemic vision</a>, together with a practical focus on concrete issues (“<i>on the <a href="http://en.wikipedia.org/wiki/Gemba">gemba</a></i>”). This is actually the core of Nassim’s Taleb “<a href="http://www.nytimes.com/2012/12/17/books/antifragile-by-nassim-nicholas-taleb.html?pagewanted=all&_r=0">Antifragile</a>” book: complex systems are learned by doing, practice is superior to thinking – and this obviously applies to architecture. Because it emphasizes practice and respect for people, lean management is geared to address the complex challenges of our environment.</span><span style="font-size: 9pt;"><o:p></o:p></span></span></div></div>

« The Lean Architect » - Do we need Abstraction and Planning « on the Gemba » ?

Today’s post is a first attempt at defining the role of a software architect in a “ Lean Software Factory ”, that is, an agile development o...

Read more »

Lean, Scrum, Agile and Extreme Programming

<div dir="ltr" style="text-align: left;" trbidi="on"><br /><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">Two thousand years ago, Epictetus told us “<b>i</b></span><b><span lang="EN-US" style="background: white; color: #222222; mso-ansi-language: EN-US; mso-bidi-font-family: Arial;">t is impossible for a<span class="apple-converted-space"> </span></span></b><em><b><span lang="EN-US" style="background-color: white;">man</span></b></em><span class="apple-converted-space"><b><span lang="EN-US" style="background: white; color: #222222; mso-ansi-language: EN-US; mso-bidi-font-family: Arial;"> </span></b></span><b><span lang="EN-US" style="background: white; color: #222222; mso-ansi-language: EN-US; mso-bidi-font-family: Arial;">to<span class="apple-converted-space"> </span></span></b><em><b><span lang="EN-US" style="background-color: white;">learn what he thinks he already knows</span></b></em><span lang="EN-US">”. This principle is still one of the main reasons why it is so difficult to deploy change, when new (development) methods are concerned. I have been advocating for lean and agile methods for many years and the most common pattern I hear is: “<i>I know, I do it already</i>” :)</span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">The goal of this post is to focus on the difference between agile in general, SCRUM in particular, and lean software development. I have been speaking about lean software development for three years and I have found that the (legitimate) confusion between lean and agile does not help. I keep hearing “<i>I already know about agile, scrum … or lean</i>”. I can relate to this. I started as a research scientist in the field of programming languages and software engineering twenty-five years ago, and was there (in the US & the academic community) when agile was born, I knew personally some the founders of the </span><a href="http://agilemanifesto.org/principles.html"><span lang="EN-US">Agile Manifesto</span></a><span lang="EN-US">. I started practicing agile and extreme programming when I returned to France to create the </span><a href="http://e-lab.bouygues.com/"><span lang="EN-US">e-Lab</span></a><span lang="EN-US">. I brewed my own agile philosophy for a small team of “PhD programmers” under the designation of </span><a href="http://organisationarchitecture.blogspot.fr/2011/12/grammaire-de-linaction-et-programmation.html"><span lang="EN-US">Fractal Programming</span></a><span lang="EN-US">. Since then, I have joined larger organizations and tried to promote agile methods. At first, I was promoting what I once knew and tried, but I have quickly discovered that the agile movement is growing constantly, and that we know much more today than what you learned yesterday. Hence the Epitectus quotation.<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US"><br /></span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">The topic of mapping the similarities and differences between lean development and agile is quite interesting because the influences and relationships make for an intricate pattern. Agile methods and its various subsequent developments have been implementing “lean principles” for many years. Lean Software Development, on the other hand, inherits from the knowledge built by agile communities. There is already a number of interesting post on this topic. </span><a href="http://www.agileweboperations.com/lean-agile-devops-related"><span lang="EN-US">Matthias Marschall’s contribution</span></a><span lang="EN-US"> is a great synthesis which is very similar to what I’ll write in a few minutes. He shows the strong relationships and the differences, and point out at the development process dimension: <i>lean</i> helps to get a larger perspective. His </span><a href="http://www.agileweboperations.com/lean-agile-devops-related"><span lang="EN-US">slideshare</span></a><span lang="EN-US"> is also interesting, but is not structured enough from my “conceptual” perspective, which I why I found useful to write my own analysis down. Another </span><a href="http://thehypertextual.com/2012/10/16/agile-et-hyperlocal-atbdx/"><span lang="EN-US">great contribution by Cecil Dijoux</span></a><span lang="EN-US"> points out the influences but stresses the benefits of lean as providing a larger perspective, especially </span><a href="http://thehypertextual.com/2012/04/25/lean-as-the-solution-to-scale-agile-methodologies/"><span lang="EN-US">the ability to scale up agile methods</span></a><span lang="EN-US"> to enterprise management and transformation. Cecil talks about </span><a href="http://www.morisseauconsulting.com/index.php/livre-kanban-pour-lit/"><span lang="EN-US">Kanban</span></a><span lang="EN-US">, to emphasize that it goes further than the traditional visual management practices of SCRUM. There are so many other examples. For instance, talks about agile methods such as </span><a href="http://fr.slideshare.net/LaurentSarrazin/2012-0525agile-francesweetrupture03final"><span lang="EN-US">Laurent Sarrazin</span></a><span lang="EN-US">’s which makes the influence of lean principles obvious. Laurent’s is a great example of a modern talk about agile, which goes much deeper that what I was hearing 15 years ago. In a reciprocate manner, great pieces about lean development, such as </span><a href="http://welovelean.com/write-up-10-things-ive-learnt-about-lean-star"><span lang="EN-US">Welovelean</span></a><span lang="EN-US">’s, are still very close to the founding principles of the Agile Manifesto.<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US"><br /></span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">The claim of this post is : <b>there is more to lean software development than a great synthesis of the best agile methods</b>. To make my point, I will briefly summarize the key ideas of the successive steps of the evolution path from </span><a href="http://en.wikipedia.org/wiki/Agile_software_development"><span lang="EN-US">Agile</span></a><span lang="EN-US">, </span><a href="http://en.wikipedia.org/wiki/Extreme_programming"><span lang="EN-US">XP</span></a><span lang="EN-US">, </span><a href="http://en.wikipedia.org/wiki/Scrum_(development)"><span lang="EN-US">SCRUM</span></a><span lang="EN-US"> to </span><a href="http://en.wikipedia.org/wiki/Lean_software_development"><span lang="EN-US">Lean Software Development</span></a><span lang="EN-US">. I need to stress out that the following description is both too brief and incomplete. It is quite succinct because (a) I have already talked about this in </span><a href="http://informationsystemsbiology.blogspot.fr/2011/11/lean-it-devops-and-cloud-programming.html"><span lang="EN-US">previous posts</span></a><span lang="EN-US"> (b) it is easy to find better references for Agile (</span><a href="http://agilemanifesto.org/principles.html"><span lang="EN-US">manifesto</span></a><span lang="EN-US">), </span><a href="http://www.extremeprogramming.org/"><span lang="EN-US">XP</span></a><span lang="EN-US">, </span><a href="http://www.scrum.org/Resources/What-is-Scrum"><span lang="EN-US">SCRUM</span></a><span lang="EN-US"> or </span><a href="http://fr.slideshare.net/jpvajda/lean-software-development-principles"><span lang="EN-US">Lean Software</span></a><span lang="EN-US">. It is incomplete by construction, since to promote clarity, I have only selected three additional ideas/principles when moving from Agile to XP, then to SCRUM, then to LSF.<o:p></o:p></span></span></div><div class="MsoNormal"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">Let me then characterize the essence of agile (when it started) as the combination of: <o:p></o:p></span></span></div><div class="MsoNormal"></div><ol style="text-align: left;"><li><a href="http://blog.extremeplanner.com/2006/08/small-teams-make-better-software.html" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Small teams</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"> : a strong emphasis on strong interactions.</span></li><li><a href="http://fr.slideshare.net/SteveRogalsky/agile101-small-batches" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Small batches</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"> : break a larger project into manageable chunks (sprints), do them one at a time and constantly re-evaluate what the next chunk will be.</span></li><li><a href="http://agile.dzone.com/articles/qa-agile-approach-0" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Time boxing</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"> : there is nothing like a time constraint to help you focus on what really matters. Deliver less, but deliver on time.</span></li><li><a href="http://www.agilemodeling.com/essays/executableSpecifications.htm" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Coevolution of code/design/architecture</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"> : the end of the waterfall model, the code and its <a href="http://www.leanessays.com/2011/08/dont-separate-design-from.html">requirements</a>are built continuously.</span></li><li><a href="http://heratech.wordpress.com/2009/12/28/agile-tenet-6%E2%80%93-face-to-face-communication/" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Role of face-to-face communication</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"> : A key insight of agile is that to break the waterfall model, you need to reintroduce face-to-face interaction and banish the email push, which breaks the responsibility and engagement chains.</span></li><li><a href="http://www.agilemodeling.com/artifacts/userStory.htm" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">User stories</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"> : the only way to capture a need is to give it some flesh and make it into a story. Read Daniel Pink’s “</span><a href="http://en.wikipedia.org/wiki/A_Whole_New_Mind" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">A Whole New Mind</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">” to understand why.</span></li></ol><br /> <div class="MsoListParagraphCxSpLast" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;">There is a lot of </span><i style="font-family: Verdana, sans-serif;">lean</i><span style="font-family: Verdana, sans-serif;"> in these principles & practices. Obviously once cannot reduce agility to this short list of principles, I should also explain a lot of associated practices (supported with tools) regarding the development process. Let me now turn to Extreme Programming, which I will characterize with three additional ideas (or increased emphasis):</span></div><div class="MsoListParagraphCxSpLast" style="text-align: justify;"></div><ol><li><a href="http://en.wikipedia.org/wiki/Test-driven_development" style="font-family: Verdana, sans-serif; text-align: left; text-indent: -18pt;"><span lang="EN-US">Test-driven development</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-align: left; text-indent: -18pt;">: the best way to develop code is to write test cases first.</span></li><li><a href="http://www.extremeprogramming.org/rules/overtime.html" style="font-family: Verdana, sans-serif; text-align: left; text-indent: -18pt;"><span lang="EN-US">Sustainable pace</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-align: left; text-indent: -18pt;">: the only way to satisfy a customer in the software world is to last, because software is a “living thing” that evolves constantly. Hence, the rhythm must be set to reach long-term goals and respect everyone in the development process.</span></li><li><a href="http://elegantcode.com/2011/01/21/book-review-extreme-programming-explained/" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Code is valuable</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">: XP was the first method to significantly raise the status of code, through practices such as code reviews, refactoring or pair programming and principles such as “elegant code”, “do it right the first time”, etc.</span></li></ol><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">Here, as well as in the following section on SCRUM, lean is everywhere </span><span lang="EN-US">J</span><span lang="EN-US"> That is, we can see these principles as the application of lean principles to software development, derived from the <a href="http://en.wikipedia.org/wiki/Toyota_Production_System">TPS</a>. I would then characterize SCRUM with the following additional contributions:<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"></div><ol><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-align: left; text-indent: -18pt;"><span style="font-size: 7pt;"> </span></span><a href="http://www.xqa.com.ar/visualmanagement/" style="font-family: Verdana, sans-serif; text-align: left; text-indent: -18pt;"><span lang="EN-US">Visual Management</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-align: left; text-indent: -18pt;">: give the team a project room, use the walls to share release notes, the backlog, planning information, etc.</span></li><li><a href="http://en.wikipedia.org/wiki/Stand-up_meeting" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Practices and Rites</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">: SCRUM is practical method that makes agile easier to learn through its rites, such as standup meetings.</span></li><li><a href="http://zacgery.blogspot.fr/2012/11/reflections-in-scrums-silver-mirror.html" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Reflection</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">: SCRUM proposes </span><a href="http://www.mountaingoatsoftware.com/scrum/sprint-retrospective/" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">sprint  retrospectives</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">, a great tool to promote continuous improvement. One could say that standing back is part of agile, but SCRUM brings a true emphasis on “doing it for real”.</span></li></ol><div class="MsoNormal"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">These twelve sets of practices make a maturity model for SCRUM. To qualify to lean software development, I would add another three steps:<o:p></o:p></span></span></div><div class="MsoNormal"></div><ol style="text-align: left;"><li><a href="http://en.wikipedia.org/wiki/Kanban_(development)" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Kanban</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">: the application of visual management to the visualization of process flow and work in process (WIP). Kanban is a critical tool to reduce WIP and to implement pull (just in time), two key lean principles.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><a href="http://en.wikipedia.org/wiki/Kaizen">Kaizen</a> :  the heart of the lean approach towards problem solving. Lean makes problem solving a team activity and a learning tool. There are many associated tools and practices such as the “<a href="http://en.wikipedia.org/wiki/5_Whys">Five whys</a>” or <a href="http://sloanreview.mit.edu/the-magazine/2009-summer/50408/toyotas-secret-the-a3-report/">Toyota’s A3</a>.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><a href="http://en.wikipedia.org/wiki/5S_(methodology)">5S</a> and <a href="http://www.allaboutagile.com/lean-principles-1-eliminate-waste/">waste removal</a> : sort, clean-up, organize, focus on simplicity and writing less code. Some of it is part of XP, <a href="http://leansoftwarepractices.com/">but lean goes further</a>, with more tools and deeper practices (e.g. <a href="http://en.wikipedia.org/wiki/Value_stream_mapping">value-stream mapping</a>).</span></li></ol><br /> <div class="MsoListParagraphCxSpMiddle" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;">This is a simplified view of lean software (although the combined practices of the 15 previous bullet points represents a fair amount of commitment), see the seven principles of </span><a href="http://www.poppendieck.com/" style="font-family: Verdana, sans-serif;">Mary and Tom Poppendieck</a><span style="font-family: Verdana, sans-serif;"> or my own description of </span><a href="http://informationsystemsbiology.blogspot.fr/2012/07/lean-software-factory.html" style="font-family: Verdana, sans-serif;">Lean Software Factory</a><span style="font-family: Verdana, sans-serif;"> for more details. A great practical introduction to lean software development is “</span><a href="http://www.crisp.se/file-uploads/Lean-from-the-trenches.pdf" style="font-family: Verdana, sans-serif;">Lean from the Trenches</a><span style="font-family: Verdana, sans-serif;">” by Henrik </span>Kniberg<span style="font-family: Verdana, sans-serif;">.</span></div><div class="MsoListParagraphCxSpMiddle" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><br /></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><b><span lang="EN-US">Another key evolution of the past 20 years is the scope of agile/lean development method, which has grown from software development to product innovation</span></b><span lang="EN-US">. In the spirit of keeping this post simple, I will consider the scope with four successively embedded steps (from the smallest to the largest):<o:p></o:p></span></span></div><div class="MsoNormal"></div><ul style="text-align: left;"><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">Software development is the original scope of agile methods. It obviously includes testing, as well as the proper code management & configuration tools to make coding as efficient as possible. See my reference to Curt Hibbs in my <a href="http://informationsystemsbiology.blogspot.fr/2012/07/lean-software-factory.html">previous post on software factories</a>.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">The next step is to move to <a href="http://en.wikipedia.org/wiki/Continuous_integration">continuous integration</a>. The ability to generate a “fresh product” every night (for instance) that can be fully tested, with as many automated test as possible, yields a completely different culture and level of innovation. This requires a significant amount of tools and training. This is the case where you need to change the tools in order to change the culture.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">The third step is <a href="http://radar.oreilly.com/2009/03/continuous-deployment-5-eas.html">continuous deployment</a>, which is best illustrated by <a href="http://en.wikipedia.org/wiki/DevOps">Devops</a>. I have mentioned Devops <a href="http://informationsystemsbiology.blogspot.fr/2011/11/lean-it-devops-and-cloud-programming.html">already</a>, the big step is to include operations in the lean/agile process. Moving from CI to CD requires more tools, more automation … and a big culture change since it require to move from “agile in the lab” to “agile in the field”.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">The last step is <a href="http://asqcincinnati.org/Library/continuousimprovement/ImprovementToInnovation.pdf">continuous product improvement and innovation</a>, which is beautifully explained in <a href="http://organisationarchitecture.blogspot.fr/2011/08/lean-startup-et-lean-product.html">The Lean Startup</a> from Eric Ries. In addition to continuous deployment, we need to gather continuous feedback (e.g., with <a href="http://en.wikipedia.org/wiki/A/B_testing">A/B testing</a>) and re-inject this knowledge into the development cycle. A great source to better understand lean startup principles is <a href="http://www.ashmaurya.com/">Ash Maurya’s blog</a>, and these principles do not apply to startups only!</span></li></ul><br /> <div class="MsoNormal" style="text-align: justify;"><b><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">Lean Software Development is the combination of both dimensions: the depth of lean practices and the scope</span></span></b><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">. I could easily derive a maturity index from this: the maturity level would be the product of:  (a) how many of the 15 principles are really in action, times (b) what is the scope of the lean/agile process. I won’t go into this direction because lean software is much more than a development method; it is a work philosophy, in the “<a href="http://en.wikipedia.org/wiki/The_Toyota_Way">Toyota Way</a>” sense. It is mostly a work philosophy that is built on the <a href="http://organisationarchitecture.blogspot.fr/2012/11/motivation-lean-et-stress.html">motivation and satisfaction</a> of the software producers. A key insight from “<a href="http://www.amazon.fr/Implementing-Lean-Software-Development-Concept/dp/0321437381">Implementing Lean Software Development</a>” is that one must move away from a project-oriented culture based on contracts towards a product-oriented culture based on collaboration. A wonderful illustration of the scope of the lean transformation is the great book “<a href="http://www.amazon.fr/Les-G%C3%A9ants-Web-Pratiques-Architecture/dp/2952589534/ref=sr_1_1?ie=UTF8&qid=1354347141&sr=8-1">Les Géants du Web</a>”, just released by <a href="http://blog.octo.com/tag/les-geants-du-web/">Octo</a>. Its subtitle is “culture / practice / architecture”; most of what I have just described is beautifully explained and illustrated. There is lean and agile everywhere in that book, but this is a book for everyone, not reserved to managers or software developers. The book is about what makes the best “Web companies” successful, from a culture, practice and architecture point of view. However, the similarity with the lean philosophy is striking.</span><o:p></o:p></span></div></div>

Lean, Scrum, Agile and Extreme Programming

Two thousand years ago, Epictetus told us “ i t is impossible for a   man   to   learn what he thinks he already knows ”. This principle is ...

Read more »

Systemic Simulation of Smart Grids : First Results

<div dir="ltr" style="text-align: left;" trbidi="on"><w:sdt contentlocked="t" id="89512093" sdtgroup="t"><span style="font-family: "Calibri","sans-serif"; font-size: 1.0pt; mso-ansi-language: FR; mso-ascii-theme-font: minor-latin; mso-bidi-font-family: "Times New Roman"; mso-bidi-language: AR-SA; mso-bidi-theme-font: minor-bidi; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: FR; mso-fareast-theme-font: minor-fareast; mso-hansi-theme-font: minor-latin;"><w:sdtpr></w:sdtpr><w:sdt docpart="60B50BC023144821ABA7290EC34FE67B" id="89512082" storeitemid="X_66CC8B3F-D460-4797-8657-E0B3B1ECE80D" text="t" title="Titre du message" xpath="/ns0:BlogPostInfo/ns0:PostTitle"></w:sdt></span> </w:sdt><br /><div class="Publishwithline"><br /></div><div style="border-bottom: solid #4F81BD 1.0pt; border: none; mso-border-bottom-themecolor: accent1; mso-element: para-border-div; padding: 0cm 0cm 2.0pt 0cm;"> <div class="underline"><br /></div></div><div class="PadderBetweenControlandBody" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span style="font-size: 10pt; line-height: 12pt; text-indent: 14.2pt;">The month of August provides me with the opportunity to get back to my computer and to resume my programming projects. I have been able to complete the first step of S3G (</span><a href="http://informationsystemsbiology.blogspot.fr/2011/09/systemic-simulation-of-smart-grids.html" style="font-size: 10pt; line-height: 12pt; text-indent: 14.2pt;">Systemic Simulation of Smart Grids</a><span style="font-size: 10pt; line-height: 12pt; text-indent: 14.2pt;">) which I started last year. I had the opportunity to attend the </span><a href="http://www.naefrontiers.org/Symposia/EU-USFOE/32025/2011EU-USFOE.aspx" style="font-size: 10pt; line-height: 12pt; text-indent: 14.2pt;">EU-US Frontiers of Engineering Symposium</a><span style="font-size: 10pt; line-height: 12pt; text-indent: 14.2pt;">, where I have presented my project and received encouraging and interesting feedback.</span></span></div><div class="MHeading2" style="line-height: 12pt; margin: 0cm 0cm 0.0001pt; text-align: justify;"><span lang="EN-US" style="font-size: 10pt;"><span style="font-family: Verdana, sans-serif;">The objective of S3G is to simulate the production and consumption of electricity throughout a long period of time (15 years). The S3G model, which is illustrated by the following figure, may be summarized with five parts:<o:p></o:p></span></span></div><div class="MHeading2" style="margin: 0cm 0cm 0.0001pt;"></div><ul style="text-align: left;"><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -18pt;">Energy demand: for each city, energy demand is generated from an hour-by-hour and day-by-day template, adding some random variation (the extent of which is a model parameter) together with a city-specific variation. This number is then reduced by the amount of “negaWatts”, computed from the total amount invested by the city. The model uses a ratio obtained from a concave-increasing function of the investment.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -18pt;">Dynamic Pricing: both suppliers and operators use a simple affine pricing model, with a constant price when the demand is less than a “base power”, and a linear formula when the demand is higher.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -18pt;">Production: suppliers use nuclear power according to planned schedule and adjust to resulting demand with fossil plants. Operators always use their green power (store it in the “buffer” or resell it when there is too much of it). They adjust to the city demand with their own fossil plant and wholesale electricity from suppliers, at the lowest marginal cost.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -18pt;"><span style="font-size: 7pt; line-height: normal;"> </span></span><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -18pt;">Consumption: The actual electricity consumption for each city is the demand, minus “shaving”, which is obtained by applying an S-curve to the sale price.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -18pt;">Market-share: for each city, the market balance between the national supplier and the local operator is determined yearly using another S-curve.</span></li></ul><br /> <div class="MHeading2" style="line-height: 12.0pt; margin-bottom: .0001pt; margin: 0cm;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyXOwvxedvl-ED4CIdQZNX6S3gAieOSMNzlkQhG6iRuWOCVs_cL9nDdUg1z0DzACJZizEmAPi8HgKQ_hPMLaeuqAt9ndyqo_MfABSmk2fTOHmBXwtEEkUQPaEpeit9gQDquQrEPeU3cd4/s1600/Diapositive1.JPG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyXOwvxedvl-ED4CIdQZNX6S3gAieOSMNzlkQhG6iRuWOCVs_cL9nDdUg1z0DzACJZizEmAPi8HgKQ_hPMLaeuqAt9ndyqo_MfABSmk2fTOHmBXwtEEkUQPaEpeit9gQDquQrEPeU3cd4/s640/Diapositive1.JPG" width="640" /></a><span style="font-family: Verdana, sans-serif;"></span></div><div style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US" style="font-size: 10pt; line-height: 12pt;">This S3G model is both simple and complex. It is simple because it is based on a handful of equations, resulting in a simulation code that is 500 lines long. On the other hand, it is a complex model for two reasons. On the one hand, there are multiple feedback and interaction loops that make it difficult to analyze how the system will react to perturbations. On the other hand, there are many unknown parameters in this model (such as market sensitivity, demand-response behavior, negaWatt capabilities, etc.).</span><span lang="EN-US" style="font-size: 10pt; line-height: 12pt;"> </span></span></div><br /><div class="MHeading2" style="line-height: 12.0pt; margin-bottom: .0001pt; margin: 0cm;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US" style="font-size: 10pt;"><br /></span></span></div><div class="MHeading2" style="line-height: 12.0pt; margin-bottom: .0001pt; margin: 0cm;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US" style="font-size: 10pt;"><br /></span></span></div><div class="MHeading2" style="line-height: 12.0pt; margin-bottom: .0001pt; margin: 0cm;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBur6yqBxsJcyAB2eYRNSmsgScw8FBuTsIMEBmmAKPFEaX-Y4DuVG9UtLoTf4s3wqnA7XMuqebL4kpT8tfuHnRUlBNeHif7ano-qquIdmkWWtCOO138KDHjnMybAqj0LDTZLOF1vAQd40/s1600/S3G-FoE-Poster-Dec12.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBur6yqBxsJcyAB2eYRNSmsgScw8FBuTsIMEBmmAKPFEaX-Y4DuVG9UtLoTf4s3wqnA7XMuqebL4kpT8tfuHnRUlBNeHif7ano-qquIdmkWWtCOO138KDHjnMybAqj0LDTZLOF1vAQd40/s640/S3G-FoE-Poster-Dec12.jpg" width="640" /></a><span style="font-family: Verdana, sans-serif;"><span lang="EN-US" style="font-size: 10pt;">The following figure shows, on the right part, a rough summary of the simulation loop that is run for each time period (3 hours, hence 8 times per day). On the left part, it shows the three main <a href="https://sites.google.com/site/yvesresearchagenda/home">GTES generic procedures</a>.<o:p></o:p></span></span></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQoHEooBvOHZnz7dU7e-xTRPQvxwC5RQUOTf9jk3kji9Oo9pR-enFz9kSUFeYuOvkN7wRSubPBlQAo8WClqhQ45GSkFev26fHMSvXZdSCOMwQfj6FQsuhQO4VnpJdCzLF0RTCyBd-qLJ0/s1600/S3G-simulation-loop.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQoHEooBvOHZnz7dU7e-xTRPQvxwC5RQUOTf9jk3kji9Oo9pR-enFz9kSUFeYuOvkN7wRSubPBlQAo8WClqhQ45GSkFev26fHMSvXZdSCOMwQfj6FQsuhQO4VnpJdCzLF0RTCyBd-qLJ0/s640/S3G-simulation-loop.jpg" width="640" /></a></div><div class="MText" style="line-height: 12pt; text-align: justify;"><span style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt;">I have already spent some time on computational experiments that will be presented at </span><a href="http://www.csdm2012.csdm.fr/" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt;">CSDM</a><span style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt;">in December. </span><span style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt;">A GTES simulation run returns the average and standard deviation of a few key business parameters, as well as some indication of the </span><a href="http://organisationarchitecture.blogspot.fr/2008/01/lapproche-gtes-et-les-quilibres-entre.html" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt;">Nash convergence</a><span style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt;">. Giving averages and a few deviations is a poor restitution of the rich data gathered during the computational experiment, but the goal is simply to “get a feeling for what is happening”, as opposed to producing a forecast. The current set of experiments is designed to understand the main issues that were exposed in </span><a href="http://informationsystemsbiology.blogspot.fr/2011/09/systemic-simulation-of-smart-grids.html" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt;">the previous post</a><span style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt;">. I have defined a fictional country somehow similar to France decomposed into 10 regions/cities. I have run 8 experiments that may be defined as follows:</span></div><div class="MText"></div><ol style="text-align: left;"><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -17.85pt;">The “default” is a reference point, from which “what-if” sensitivity analysis is made. The economic parameters are set in such a way that alternate operators start with a 20% market-share and should be able to increase it if they demonstrate a better management of variability.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -17.85pt;">The second experience raises the variability of energy consumption (globally), while the third experience raises the local variability (each city is more different from each other).</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -17.85pt;">The fourth experiment doubles the fossil energy price (gas and coal). In the default scenario, it is randomly drawn between 20€ and 40€/MWh.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -17.85pt;">The fifth experiment imposes a 5% reduction of the nuclear assets for the supplier during the first 5 years.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -17.85pt;">The sixth experiment sets a carbon tax at 100€/t, the proceeds of which is used by the “regulator” to subsidize green energy investment.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -17.85pt;">The seventh experiment explores the impact of overall demand variation in the next 15 years. The model assumes a constant growth/decline of electricity demand which is expressed as a percentage. This experiment plays with different values to see if <i>demand impacts the profitability of smart grids</i>.</span></li><li><span style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -17.85pt;">The last experiment is a small variation of the first one, where wholesale prices are more rigidly constrained. During this set of experiment, I considered that the constraints that regulate wholesale prices for the supplier are fixed. An interesting next step will be </span><i style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -17.85pt;">to make them a strategic parameter for the regulator</i><span style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -17.85pt;"> (which is a better representation of reality and makes for a more interesting game).</span></li></ol><div style="text-indent: -23.799999237060547px;"><span style="font-family: Verdana, sans-serif; font-size: x-small;"><span style="line-height: 16px;"><br /></span></span></div><span style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt;"><div style="text-align: justify;"><span style="font-size: 10pt; line-height: 12pt;">I still need to do some additional tuning before presenting my results at CSDM. I will then make my slides available on this blog. For once, I </span><i style="font-size: 10pt; line-height: 12pt;">need to run longer simulations</i><span style="font-size: 10pt; line-height: 12pt;"> (with larger samples) to ensure that the Monte-Carlo sampling is stable. I also want </span><i style="font-size: 10pt; line-height: 12pt;">to introduce better local optimization meta-heuristics</i><span style="font-size: 10pt; line-height: 12pt;">, such as Tabu search. One of the key issues for smart grid is the pricing model, both for the operator and the supplier. The more complex the pricing model, the better the actor may play with the </span><a href="http://en.wikipedia.org/wiki/Demand_response" style="font-size: 10pt; line-height: 12pt;">demand-response</a><span style="font-size: 10pt; line-height: 12pt;">concept. However, a more complex princing model requires better optimization techniques to ensure that the automatic computation from GTES (where each actor “learns” its “best response”) is relevant.</span></div></span><br /> <div class="MText" style="line-height: 12.0pt;"><br /></div><div class="MText" style="line-height: 12.0pt; text-indent: 0cm;"><span lang="EN-US" style="font-size: 10.0pt; mso-fareast-language: EN-US;"><span style="font-family: Verdana, sans-serif;">This being said, here are a few findings that may be drawn from the hundreds of runs made with the S3G model, and that are worth sharing: <o:p></o:p></span></span></div><div class="MText" style="text-indent: 0cm;"></div><ul style="text-align: left;"><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -17.85pt;"><span style="font-size: 7pt; line-height: normal;"> </span></span><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -17.85pt;">There is a systemic benefit of distribution and autonomy to cope with variation. This is shown in our second and third line. It is a “subtle” variation (small effects), which means that the economy of the local operator is dominated by its capacity to operate at much lower customer management costs than the supplier.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -18pt;">CO2 tax increases play a very small role, and one that is difficult to anticipate since it both favors the local operator (support green subsidies) and the supplier (raises the difference between fossil and nuclear).</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -18pt;">“De-nuclearization” is a favorable scenario for smart grid operators, as are most regulations that are adverse to the supplier. The obvious limitation is the resulting price increase that reduces the total economy output (and the country’s competitiveness).</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -18pt;">The “community advantage” (that is, the ability for a local operator to better manage the demand-response loop because it is “closer” to its end customer) is marginal, and it is quite unclear if the payback from demand-response management is enough to sustain the operator’s business model.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -18pt;">Investing in local storage is never an interesting option (at current prices). We needed to slash the price by over an order of magnitude to see a viable payback in less than 10 years.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -18pt;">There is a clear competition between local operators and suppliers. The learning component of GTES makes for “agile” players who react closely to each other signals. The pricing structure plays an important role (we have only explored a simple variable pricing scheme). A logical consequence is the importance of regulation.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: 10pt; line-height: 12pt; text-indent: -18pt;">The results are sensitive to the strategies of the player. <i>A next step for S3G is to build a “strategy matrix”</i>, which is a tabular “what-if” sensitivity analysis where we see what happens if the goals of the players are changed.</span></li></ul><br /> <div class="MsoNormal"><br /></div></div>

Systemic Simulation of Smart Grids : First Results

The month of August provides me with the opportunity to get back to my computer and to resume my programming projects. I have been able t...

Read more »

Lean Software Factory

<div dir="ltr" style="text-align: left;" trbidi="on"><br /><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US">This post is a follow-up to an invited talk that I delivered at </span><a href="http://www.cesames.net/soiree-thematique-du-20-mars-2012-6844"><span lang="EN-US">CESAMES</span></a><span lang="EN-US">. That presentation, which is available on the box.net widget on the left, talked about complexity, architecture and lean software development. It started from the question – its title – “<i>How to design information systems</i>?” and ended with an introduction to </span><a href="http://informationsystemsbiology.blogspot.fr/2011/11/lean-it-devops-and-cloud-programming.html"><span lang="EN-US">lean software development</span></a><span lang="EN-US">. Today I’ll go further and explore the relationship between the (software) product and the (software) factory. The key idea is derived from </span><a href="http://en.wikipedia.org/wiki/The_Toyota_Way"><span lang="EN-US">the Toyota Way</span></a><span lang="EN-US">: <b>you need a great factory to manufacture a great product</b>. Not surprisingly this will lead to the concept of “Lean Software Factory”. This post is organized as follows. The first section talks about what makes a good software product. I will focus mostly here on software development, but most of what will follow is equally relevant to information systems. The second section deals with the concept of “</span><a href="http://en.wikipedia.org/wiki/Software_factory"><span lang="EN-US">software factory</span></a><span lang="EN-US">”. Looking at the “software factory” means taking into account the software manufacturing process into account, and not only the final product. It is also a metaphor that carries the need for automation, engineering methods and industrial discipline. The last section is about “<b>lean software factories</b>”, which add agile development practices and wisdom to the strengths of software factories, together with a continuous improvement and culture geared towards excellence that is inherited from <i>lean management</i>.<o:p></o:p></span></div><div class="MsoNormal"><span lang="EN-US"><br /></span></div><div class="MsoNormal"><b><span lang="EN-US" style="font-size: 14.0pt; mso-ansi-language: EN-US; mso-bidi-font-size: 10.0pt;">1. What makes a good SW product?<o:p></o:p></span></b></div><div class="MsoNormal"><b><span lang="EN-US" style="font-size: 14.0pt; mso-ansi-language: EN-US; mso-bidi-font-size: 10.0pt;"><br /></span></b></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US">There are many expected qualities and assessing a piece of software is a multi-dimensional tasks. Here I will only focus on three of them, because they are both very popular (all books and papers related to software engineering discuss those three) and very important to the value that is produced through a software product:<o:p></o:p></span></div><div class="MsoNormal"><span lang="EN-US"><br /></span></div><div class="MsoNormal"></div><ul style="text-align: left;"><li><a href="http://en.wikipedia.org/wiki/Modularity" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">Modularity</span></a><span lang="EN-US" style="background-color: white; text-indent: -18pt;"> – “the degree to which a system’s components may be separated and recombined” (Wikipedia) … or the capacity of the architecture to maximize in its decomposition the independence of subcomponents. Modular systems are easier to maintain and to evolve. In the world of software, modularity translates into the minimization of “impacts” when something must be changed.</span></li><li><a href="http://en.wiktionary.org/wiki/versatility" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">Versability</span></a><span style="background-color: white; text-indent: -18pt;"> </span><span lang="EN-US" style="background-color: white; text-indent: -18pt;">/ </span><a href="http://en.wikipedia.org/wiki/Evolvability" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">evolvability</span></a><span lang="EN-US" style="background-color: white; text-indent: -18pt;"> – “the property of having many abilities”, that is software that can serve many purposes, together with “the capacity of adaptative evolution” - software may evolve rapidly to support new functions. This combination is often coined as “agility” in the software world.</span></li><li><span lang="EN-US" style="background-color: white; text-indent: -18pt;"><span style="font-size: 7pt;"> </span></span><a href="http://en.wikipedia.org/wiki/Openness" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">Openness</span></a><span style="background-color: white; text-indent: -18pt;"> </span><span lang="EN-US" style="background-color: white; text-indent: -18pt;">: for a piece of software, being open may mean many things. It may means that it exposes API (<i>application programming interfaces</i>) so that other software components may interact with it. It may also mean that the code is open to all for scrutiny and collaborative improvement, as in “open source”. It may mean that the piece of software is built as a <b>platform</b>, which is designed to be enriched by subcomponents written and proposed by other developers. Writing software as a platform is at the heart of successful development strategies, such as Google’s or Facebook’s.</span></li></ul><br /> <div class="MsoNormal"><span lang="EN-US">These three qualities form a deeply interrelated network (each influences the other). They are also notoriously difficult to achieve, especially in a rapidly changing environment. These qualities are resulting properties of both:<o:p></o:p></span></div><div class="MsoNormal"><span lang="EN-US"><br /></span></div><div class="MsoNormal"></div><ul style="text-align: left;"><li><b style="background-color: white; text-indent: -18pt;"><span lang="EN-US">Software architecture</span></b><span lang="EN-US" style="background-color: white; text-indent: -18pt;">, that is, design properties which explain how the software product is organized. This part is well understood, albeit difficult: this </span><a href="http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0471228869.html" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">the art of software architecture</span></a><span lang="EN-US" style="background-color: white; text-indent: -18pt;">. If I follow </span><a href="http://en.wikipedia.org/wiki/Fran%C3%A7ois_Jullien" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">François Jullien’s</span></a><span lang="EN-US" style="background-color: white; text-indent: -18pt;"> distinction between </span><a href="http://books.google.fr/books/about/A_Treatise_on_Efficacy.html?id=OS_YqZhWuAsC&redir_esc=y" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">Greek and Chinese strategy</span></a><span lang="EN-US" style="background-color: white; text-indent: -18pt;">, architecture is clearly the Greek, top-down, left brain, goal-oriented part of the strategy for producing modular, agile and open software.</span></li><li><b style="background-color: white; text-indent: -18pt;"><span lang="EN-US">The way the product is made</span></b><span lang="EN-US" style="background-color: white; text-indent: -18pt;"> (the software development culture). This translates into “genes” which are embedded into the software product. According to the same distinction, software development culture is the Chinese, bottom-up, right brain, emergent strategy for producing modular, agile and open software. This is less understood and even less formalized, it is a form of wisdom transmitted in an oral tradition. It is also controversial since many still believe that once the specification is completed, it doesn’t matter who writes the code …</span></li></ul><br /> <div class="MsoNormal"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaQcd52wS-vkfUl_7eDwbmhp-eCx58VDfcb2-sugzA9wIsDBSdJIUgZN4bUEQkRMbhMZi9gH_y_DbWnJJYpHkFc0SIhNZZOsETOO1tCZSjtV-vyV5c6QxquIGaTLILKDXID7fQASLHOPQ/s1600/Urbanisation4e.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhaQcd52wS-vkfUl_7eDwbmhp-eCx58VDfcb2-sugzA9wIsDBSdJIUgZN4bUEQkRMbhMZi9gH_y_DbWnJJYpHkFc0SIhNZZOsETOO1tCZSjtV-vyV5c6QxquIGaTLILKDXID7fQASLHOPQ/s200/Urbanisation4e.jpg" width="140" /></a><span lang="EN-US"></span></div><div style="text-align: justify;"><span lang="EN-US" style="background-color: white;">I am certainly not one of them </span><span lang="EN-US" style="background-color: white; font-family: Wingdings;">J</span><span lang="EN-US" style="background-color: white;">. On the contrary, I have found over the years that <b>great software products most often come from companies with great software cultures</b>. It is not a new thought either, even for me. In my first book “<a href="http://www.dunod.com/informatique-multimedia/systemes-dinformation-et-reseaux/strategie-et-systemes-dinformation/manageme/urbanisation-soa-et">Urbanisation, SOA & BPM</a>”, I noticed that modularity and agility are properties that describe the whole lifecycle and not only the development part. Agility in code writing does not do much for you if you cannot run your tests in an agile mode (Chapter 6). Similarly, modularity must translate into modular testing (running fewer tests), modular deployment (deploying fewer components) and modular maintenance (fixing problems faster because the area of investigation is limited to a smaller area).</span></div><br /> <div class="MsoNormal" style="text-align: justify;"><span lang="EN-US">The remainder of this post is devoted to « <i>the way software products are built</i> », for lack of time but mostly because my previous writings and books have been about software and information systems <i>architecture</i>. You may read the CESAMES slides to get an overview on achieving modularity, agility and openness through architecture. For those of you who don’t read French, here is a “101” summary:<o:p></o:p></span></div><div class="MsoNormal"><span lang="EN-US"><br /></span></div><div class="MsoNormal"></div><ol style="text-align: left;"><li><span lang="EN-US" style="background-color: white; font-family: Symbol; text-indent: -18pt;"><span style="font-family: 'Times New Roman'; font-size: 7pt;"> </span></span><a href="http://en.wikipedia.org/wiki/Model-driven_architecture" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">Model-driven Architecture</span></a><span lang="EN-US" style="background-color: white; text-indent: -18pt;"> : architecture should be based on models and part of the development should be derived with semi-automated methods. Models are another form of “genes” for a software product: careful design at the model stage goes a long way to embed “future-proofness” (using scenario/whatif methods)</span></li><li><a href="http://www.1keydata.com/datawarehousing/data-modeling-levels.html" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">Data models</span></a><span lang="EN-US" style="background-color: white; text-indent: -18pt;"> are critical: from semantics, conceptual analys, ontology to object lifecycle management. Data architecture is the most useful tool to prevent data distribution difficulties which plague most large-scale, heterogeneous information systems.</span></li><li><a href="http://hendryluk.wordpress.com/2009/08/17/software-development-fundamentals-part-2-layered-architecture/" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">Layered Architecture</span></a><span lang="EN-US" style="background-color: white; text-indent: -18pt;"> : defines abstraction levels and reduces the management complexity, a time-proven “good practice” of software architecture.</span></li><li><a href="http://informationsystemsbiology.blogspot.fr/2009/05/soa-tale-of-two-cities.html" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">Service-Oriented Architecture</span></a><span lang="EN-US" style="background-color: white; text-indent: -18pt;">: defines the service structure so that reuse and sharing is made easier. SOA is also the proper tool to align the different stakeholders so that software fits the enterprise’s strategy. A service architecture is a reification of functional architecture (a service may be seen as the combination of a function – what it does- an interface – how to invoke it – and a contract – so-called “non-functional” properties –), which embodies abstraction and encapsulation.</span></li><li><a href="http://en.wikipedia.org/wiki/Process_architecture" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">Process Architecture</span></a><span lang="EN-US" style="background-color: white; text-indent: -18pt;"> : defines the “composition grammar” through a recursive/fractal analysis of interactions. Process architecture is to “Business Process Engineering” what SOA is to “writing down a list of exposed services”: there is much more than listing and formalizing, it is about identifying patterns and sub-patterns, and extracting those patterns which will prove stable through time. It is more an art than a science.</span></li><li><a href="http://en.wikipedia.org/wiki/Event-driven_architecture" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">Event-Oriented Architecture</span></a><span lang="EN-US" style="background-color: white; text-indent: -18pt;">: identifies “events” as pivots for interaction (which is a subpart of process architecture and a foundation for SOA). </span><a href="http://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">Publish/Subscribe</span></a><span lang="EN-US" style="background-color: white; text-indent: -18pt;"> is another time-proven pattern for building modular designs, which is built on top of a shared/structured event catalog.</span></li></ol><br /> <div class="MsoNormal"><br /></div><div class="MsoNormal"><b><span lang="EN-US" style="font-size: 14.0pt; mso-ansi-language: EN-US; mso-bidi-font-size: 10.0pt;">2. Software Factories<o:p></o:p></span></b></div><div class="MsoNormal"><b><span lang="EN-US" style="font-size: 14.0pt; mso-ansi-language: EN-US; mso-bidi-font-size: 10.0pt;"><br /></span></b></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US">Focusing on the way software products are build leads logically to the concept of “Software factory”. The following illustration shows three way of looking at a software system (or an information system):<o:p></o:p></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US"><br /></span></div><div class="MsoNormal" style="text-align: justify;"></div><ul><li><span lang="EN-US" style="background-color: white; text-align: left; text-indent: -18pt;">The first step (left-most) is to look at the technical system (leading to what we call IT : Information Technology). This view is concerned with practical components such as databases, software code and computing resources. Processes are defined in the “computer science sense”.</span></li><li><span lang="EN-US" style="background-color: white; text-align: left; text-indent: -18pt;">Then we step back and add the human into the picture, mostly the user but also other stakeholders (management, administration, strategy,…), leading to Information Systems. </span><a href="http://blogs.cio.com/chris_potts/information_systems_versus_information_technology_how_much_do_people_care" style="background-color: white; text-align: left; text-indent: -18pt;"><span lang="EN-US">IT/IS distinction</span></a><span lang="EN-US" style="background-color: white; text-align: left; text-indent: -18pt;"> is well established in information systems literature. This view adds – rightly so – usage as a key dimension (following the very profound statement that a software application that is not used is of little value), and shared enterprise artifacts such as models and business processes.</span></li><li><span lang="EN-US" style="background-color: white; text-align: left; text-indent: -18pt;">The topic of this post is to step back another time and look at the “production process” (the “factory”) that continuously builds and update the information system. On the right-most part of the illustration, we see both the information systems and the organization (usually the ISD – Information System Division in a large company) that builds and runs it. I have separated the part that builds (dev: software development) and the part that runs (op: operation) as a forward reference to </span><a href="http://en.wikipedia.org/wiki/DevOps" style="background-color: white; text-align: left; text-indent: -18pt;"><span lang="EN-US">devops</span></a><span lang="EN-US" style="background-color: white; text-align: left; text-indent: -18pt;">.</span></li></ul><br /> <div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinOq-RpWb-HvrEphU4-KBH_KdMnmAsdY9fldI9k7C6zjFhLE1BiQbnu2obL7RormrCp8D8jewKdWPPBC0ZBYguNYKAt8R-jpKrGTlJAARgmhlowQ2lcOAzbT7jpIp2xQw_0GGr0x7RWGg/s1600/SoftwareFactorySchema.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinOq-RpWb-HvrEphU4-KBH_KdMnmAsdY9fldI9k7C6zjFhLE1BiQbnu2obL7RormrCp8D8jewKdWPPBC0ZBYguNYKAt8R-jpKrGTlJAARgmhlowQ2lcOAzbT7jpIp2xQw_0GGr0x7RWGg/s640/SoftwareFactorySchema.jpg" width="640" /></a></div><div class="MsoNormal"><br /></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US">Because a software factory handles ideas and digital information, it could be argued that it is an </span><a href="http://en.wikipedia.org/wiki/Autopoiesis"><span lang="EN-US">autopoietic system</span></a><span lang="EN-US"> (this is not a car factory, there is endless supply of ideas and disk space to write new code). Autopoiesis represents the capacity of a living system to continuously develop itself and to transform the process that creates the system and which is a part of the system itself. Autopoiesis generates emergence (of properties) and (what may be seen as) auto-organization. This thread of thought is quite interesting, because it helps understanding the emergence of our three desired properties, however I will leave it to another day, to avoid adding confusion to conceptual complexity.<o:p></o:p></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US">To return to more practical grounds, the goal of a software factory is to deliver “good software products”, rapidly and efficiently. Because we now have the advantage of having read “</span><a href="http://theleanstartup.com/"><span lang="EN-US">The Lean Startup</span></a><span lang="EN-US">” (if you have not, I would urge you to do so, it’s the best way to understand what is truly expected of a great software factory), we know that there is another dimension to SW factory performance: <b>the ability to listen carefully to the product’s users and react rapidly to their expectation</b>. This is why </span><a href="http://en.wikipedia.org/wiki/Eric_Ries"><span lang="EN-US">Eric Ries</span></a><span lang="EN-US"> advocates for </span><a href="http://en.wikipedia.org/wiki/Minimum_viable_product"><span lang="EN-US">MVP</span></a><span lang="EN-US"> (Minimum Viable Products): the SF (software factory) must be able to crank up simple products very rapidly, which are exposed to customers, and modified according to their reaction (this loop is shown on the illustration).<o:p></o:p></span></div><div class="MsoNormal"><span lang="EN-US">A lot of things are known about how to build a “good” SF. Sharing a reference with the previous post, I picked the following recommendations in Curt Hibb’s book “</span><a href="http://shop.oreilly.com/product/9780596517311.do"><span lang="EN-US">The Art of Lean Software Development</span></a><span lang="EN-US">”:<o:p></o:p></span></div><div class="MsoNormal"><span lang="EN-US"><br /></span></div><div class="MsoNormal"></div><ul style="text-align: left;"><li><span lang="EN-US" style="background-color: white; font-family: Symbol; text-indent: -18pt;"><span style="font-family: 'Times New Roman'; font-size: 7pt;"> </span></span><span lang="EN-US" style="background-color: white; text-indent: -18pt;">“Practice 0” : Source Code Management and Scripted Built. This is where the word “factory” becomes self-evident: one must use tools (such as source code management) and industrial discipline to manufacture software. There is a “Maslow pyramid” of skills necessary to reach continuous deployment, scripted built is the first step.</span></li><li><span lang="EN-US" style="background-color: white; font-family: Symbol; text-indent: -18pt;"><span style="font-family: 'Times New Roman'; font-size: 7pt;"> </span></span><span lang="EN-US" style="background-color: white; text-indent: -18pt;">“Practice 1” : Automated Testing, because testing is crucial for quality, as in any industrial process (automated testing means more testing), but also because we want to run tests as early as possible, </span><a href="http://informationsystemsbiology.blogspot.fr/2011/11/lean-it-devops-and-cloud-programming.html" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">following the principles of extreme programming</span></a><span lang="EN-US" style="background-color: white; text-indent: -18pt;">.</span></li><li><span lang="EN-US" style="background-color: white; font-family: Symbol; text-indent: -18pt;"><span style="font-family: 'Times New Roman'; font-size: 7pt;"> </span></span><span lang="EN-US" style="background-color: white; text-indent: -18pt;">“Practice 2” : Continuous Integration, which means that the product is built continuously in an iterative manner, instead of the traditional waterfall model. Continuous integration to perform integration testing as early as possible; it is also a way to “see the product” and ensure that the factory is united on a common goal.</span></li><li><span lang="EN-US" style="background-color: white; font-family: Symbol; text-indent: -18pt;"><span style="font-family: 'Times New Roman'; font-size: 7pt;"> </span></span><span lang="EN-US" style="background-color: white; text-indent: -18pt;">“Practice 5” : Customer Participation, which is what makes the continuous improvement loop that is pictured in the previous figure possible. Customer participation is at the heart of </span><a href="http://en.wikipedia.org/wiki/Agile_software_development#Agile_Manifesto" style="background-color: white; text-indent: -18pt;"><span lang="EN-US">agile methods</span></a><span lang="EN-US" style="background-color: white; text-indent: -18pt;">.</span></li></ul><br /> <div class="MsoNormal" style="text-align: justify;"><span lang="EN-US">There is much more in Hibb’s book, as we shall later see, but this is a set of sound, time-proven practices. Another great source of inspiration for designing a software factory is the </span><a href="http://informationsystemsbiology.blogspot.fr/2011/11/lean-it-devops-and-cloud-programming.html"><span lang="EN-US">previously-mentioned</span></a> <a href="http://devops.com/"><span lang="EN-US">DevOps</span></a> <span lang="EN-US">“method”. Many excellent software companies, such as </span><a href="http://www.fabernovel.com/fr/works/298-facebook-the-perfect-startup"><span lang="EN-US">Facebook</span></a><span lang="EN-US">, are making references to DevOps, which is not a coincidence. Following </span><a href="http://blog.octo.com/tag/devops/"><span lang="EN-US">Octo’s lead</span></a><span lang="EN-US">, I would summarize DevOps’ contributions with two principles: “Infrastructure as code” and “Continuous deployment”. </span><a href="http://www.agileweboperations.com/the-implications-of-infrastructure-as-code"><span lang="EN-US">Infrastructure as code</span></a><span lang="EN-US"> is the ability to address all computing resources (either cloud computing or a regular data center) through API and software. It follows the principles of “Practice 0” (tools & automation) and give true flexibility and agility to the software factory. </span><a href="http://blog.xebia.fr/2012/01/09/mise-en-place-d-une-organisation-devops/"><span lang="EN-US">It also changes the “ops culture”</span></a><span lang="EN-US"> and prepares for the next step. </span><a href="http://en.wikipedia.org/wiki/Continuous_delivery"><span lang="EN-US">Continuous delivery</span></a><span lang="EN-US"> is the ultimate step of continuous development/testing/integration/deployment. It helps shortening the continuous improvement loop from Eric Ries and is necessary to move towards a “lean software factory” as we shall see later. </span><a href="http://royal.pingdom.com/2010/06/18/the-software-behind-facebook/"><span lang="EN-US">Taking a look at Facebook development and operations’ culture is worth the effort</span></a><span lang="EN-US">. Facebook has developed large-scale continuous deployment into an art. I recommend watching Chuck Rossi’s tech talk on “</span><a href="https://www.facebook.com/video/video.php?v=10100259101684977&oid=9445547199&comments"><span lang="EN-US">Pushing millions of lines of code five days a week</span></a><span lang="EN-US">”.<o:p></o:p></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US"><br /></span></div><div class="MsoNormal"><b><span lang="EN-US" style="font-size: 14.0pt; mso-ansi-language: EN-US;">3. Lean Software Factory<o:p></o:p></span></b></div><div class="MsoNormal"><b><span lang="EN-US" style="font-size: 14.0pt; mso-ansi-language: EN-US;"><br /></span></b></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US">The idea of a “lean software factory” comes from mixing “lean software development” (cf. <a href="http://informationsystemsbiology.blogspot.fr/2011/11/lean-it-devops-and-cloud-programming.html">previous post</a>) with “software factory”. That is, adding to what we just saw a few other principles which are taken from Agile methods and from the Toyota Production System (cf. <a href="http://shop.oreilly.com/product/9780596517311.do">Hibb’s book</a>). This does not mean that all projects from a lean software factory should follow a SCRUM method (take as an illustrative example of agile method). Although this is debatable, I believe that some projects are still better run with a classical V-cycle model (<a href="http://blog.ippon.fr/2012/04/25/scrum-breakfast-lagilite-chez-google/">I am not not the only one</a>). There is no magical formula to define the “SCRUM/ agile boundary”, but V-cycle still makes a lot of sense for “back-office projects” with stable and complex specifications (such as a billing system). On the other side, agile methods are better suited to front-office projects (when the human in the loop plays a bigger role), where detailed requirements are difficult to produce because the need is ill-defined or changing rapidly.<o:p></o:p></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US"><br /></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US">To keep this post short, I will summarize the concept of “Lean Software Factory” with four “principles”. Each of them would require a full post, which I may do later on – <b>Each “principle” is in fact a set of practices since “<i>lean</i> is something you do, not something you talk about”</b>.  These principles add to the previously developed concept of “Software Factory”. The first principle captures the essence of “agile methods” and, therefore, only applies to agile projects, whereas the rest applies to both forms of software development (that is, including V-cycle development).<o:p></o:p></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US"><br /></span></div><div class="MsoNormal" style="text-align: justify;"></div><ol><li><span lang="EN-US" style="background-color: white; text-align: left; text-indent: -18pt;">The first principle is to <b>apply an agile method </b>such as SCRUM (Backlog + small batch + test as early as possible + continuous build) whenever the end-user has an important stake with the software product. Agile methods fit nicely with the principles of a SF as described in the previous section. They emphasize customer participation and continuous integration. The lean “touch” adds the concept of small batches (called sprints in SCRUM) and the principle of testing as early as possible, with the goal of producing the right code on the first time. Building a product one small step at a time with regular opportunity to adapt to environment feedback is a key idea which is well developed in “The Lean Startup”. Another key inspiration from <i>lean</i> is to reduce “lead time”, which translates into short sprints but also into time-constrained steps, including design.  This last technique also works within V-cycle: using time-constrained workshops to jointly achieve the requirements and design phases. Those familiar with Google’s software development values will recognize the “lean influence” (develop quickly, only what is necessary, remove waste).</span></li><li><span lang="EN-US" style="background-color: white; text-align: left; text-indent: -18pt;">The second principle is to <b>emphasize autonomous teamwork</b>. Teamwork is critical for all kinds of projects and for most activities. A key teamwork moment from the lean culture is <a href="http://en.wikipedia.org/wiki/Kaizen">Kaizen</a>, which is used to solve problems and search for continuous improvements (two sides of the same coin). Following “the Toyota Way”, all ideas are welcome and everyone’s contribution is useful in a LSF.  There is a lot to be said about kaizen (which I won’t do today) : it goes further than the continuous improvement culture that one may find everywhere (such as the feedback loop in the SCRUM method). “Lean culture” brings a systemic approach to problem solving, including the search for root causes with “<a href="http://en.wikipedia.org/wiki/5_Whys">5 Whys</a>” and the construction/follow-up action plans, such as the famous “<a href="http://www.thequalityportal.com/notes/A3Shook.pdf">A3</a>” from Toyota. Autonomy is a systemic condition for kaizen in particular, but more generally empowerment is necessary to achieve performance.  One of the best book to understand kaizen and autonomous teams is “<a href="http://www.lean.org/Bookstore/ProductDetails.cfm?SelectedProductId=261">The Lean Manager</a>” by F. & M. Ballé.</span></li><li><span lang="EN-US" style="background-color: white; text-align: left; text-indent: -18pt;">The third principle is to <b>leverage visual management</b>, which often takes the form of a project/product room. Visual Management has multiple benefits, the first of them is <i>synchronicity</i>, that is working with the same time. In the world of an industrial factory, the lean concept that embodies synchronicity is “<i>takt time</i>”. Although <a href="http://en.wikipedia.org/wiki/Takt_time">takt time</a> also plays a role in a software factory, synchronicity is mostly about working at the same pace, without waiting. Dynamic wall-planning is the best tool to ensure synchronicity, together with the frequent stand-up meetings in front of the said planning, where the whole team keeps track of progress and difficulties. Visual management is also a tool for systemic training. Toyota encourages its workers to use as many schematics and illustrations as needed to share the detailed functioning of technical processes or pieces of equipment. This translates easily into the world of software. Using the walls to share architectural insights with the team is an efficient way to contribute to continuous learning as well as increase efficiency and prevent errors. Visual management is also used to visually control “work in process” (<a href="http://en.wikipedia.org/wiki/Work_in_process">WIP</a>). Reducing WIP is another key tenet from <i>lean management</i>that applies very nicely to software development. The practice of visualizing WIP with small cards (post-it) that are pasted on the wall according to their status in the development process is called <i>Kanban</i>in the lean IT world, in reference <a href="http://en.wikipedia.org/wiki/Kanban">to the Totota pull system</a>. Kanban / JIT (just-in-time) in the world of software is not as strictly enforced as in the world of industrial manufacturing. Still, the “pull” chain of control, where every step in a process is “serving” the following step (that is, providing what is needed at the right time, according to the availability of the receiving step), is a great tool to increase efficiency and avoid “overload” of tasks that are waiting to get completed. The “kanban” practice has two benefits: making each member of the team aware of the availability of others who will use their work later on, and reducing overloads and piles of unfinished work through an implicit capacity limit to what is accepted into the development process. The emphasis is on “no waiting” to both improve agility and quality (asynchrony is the best way to lose “context” when the control is handled from one player to another and to produce misunderstandings – this is explained in <a href="http://www.dunod.com/entreprise-gestion/entreprise-et-management/strategie-et-politique-de-lentreprise/processus-et-entreprise-20">my last book</a>, and is especially true for software development, which is a complex task, and where one quickly looses focus once your work  is completed.</span></li><li><span lang="EN-US" style="background-color: white; text-align: left; text-indent: -18pt;">The last principle tells that a Lean Software Factory is a place with “<b>love for the product</b>”. This supposes two things: to know, understand and “love” the customer, on one hand, and to love software code on the other hand. The love of code is another multi-dimensional mix of culture and practice.  The practice part draws on extreme programming (code reviews, peer writing) and software factory (cf. previous section) with a “lean touch”. Mary Poppendieck explains how the <a href="http://en.wikipedia.org/wiki/5S_(methodology)">5S</a>of lean may be applied to the practice of source code management. Lean practice also translates in writing less code (see previously mentioned Curt Hibb’s book). The culture part is also a mix of aesthetic (writing elegant, minimal code that one is pleased to show to the other members of the team) and discipline (following coding styles and guidelines, appropriate density of comments, etc.). This is very close to the values of open source software, one could say that a lean software factory should write its code as if all of it was going to be open source. We are getting back to the opening argument in the introduction: designing open software is a state of mind, hence it should be supported by the culture. There are two reasons why “the love of code” is emphasized as a key value. The first one is that application of what is learned about Toyota: to build a great company, the employees need to love the product that they are making. For Toyota it is about cars, in a software factory, it is about code. Lean management is all about professional pride, technical skills and craft. A lean software factory is not about undistinguishable “man-days” that may be outsourced to a lower-cost country. The second reason is that code is a “live object”, with a renewal rate which is getting higher as the world becomes more complex (<a href="http://blog.ippon.fr/2012/04/25/scrum-breakfast-lagilite-chez-google/">Petra Cross from Google says that 50% of their code changes every month</a>). Code that grows iteratively needs to be re-factored regularly (iteration yields accumulation); like a garden, it needs constant care and attention.</span></li></ol><br /> <div class="MsoListParagraphCxSpLast"><br /></div><div class="MsoNormal"><br /></div></div>

Lean Software Factory

This post is a follow-up to an invited talk that I delivered at CESAMES . That presentation, which is available on the box.net widget on the...

Read more »

Lean IT, Devops and Cloud Programming

<div dir="ltr" style="text-align: left;" trbidi="on"><br /><div class="MsoNormal"><span style="font-family: 'Trebuchet MS', sans-serif;"><span lang="EN-US">I have become more and more interested with lean IT over the years. It started with the book "The Art of Lean Software Development" by Curt Hibbs. </span></span></div><div class="MsoNormal"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGEcuxKq3vWjNUEwOE0Cv0mMsZX0d8BmovHjdBh5mM2eoWyYFQfmMk0V91Yhr4uncaN-SNKQT1bvWHSG4F27mCUbWFu2AFjyTcBHmRQGsXyHzuYeGjLWIdghXk5IASuwGTUa290GCQbpM/s1600/ArtLeanSoftwareDevelopment.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGEcuxKq3vWjNUEwOE0Cv0mMsZX0d8BmovHjdBh5mM2eoWyYFQfmMk0V91Yhr4uncaN-SNKQT1bvWHSG4F27mCUbWFu2AFjyTcBHmRQGsXyHzuYeGjLWIdghXk5IASuwGTUa290GCQbpM/s1600/ArtLeanSoftwareDevelopment.jpg" /></a><span style="font-family: 'Trebuchet MS', sans-serif;"><span lang="EN-US">I enjoyed this book because its introduction of lean is 100% compatible of <a href="http://organisationarchitecture.blogspot.com/2007/12/le-lean-cest-quoi.html">what I learned</a> from Jeffrey Liker and other great authors about <a href="http://en.wikipedia.org/wiki/Toyota_Production_System">TPS</a></span><span lang="EN-US">. This simple book helped me draw the lines between good software development methods such as <a href="http://www.extremeprogramming.org/">extreme programming</a></span><span lang="EN-US"> or <a href="http://en.wikipedia.org/wiki/Agile_software_development#Agile_Manifesto">agile programming</a>, and lean management. For those of you who are not familiar with lean or extreme programming, here is a very crude summary of some of the most salient points of lean software development :<o:p></o:p></span></span></div><div class="MsoNormal"></div><ul style="text-align: left;"><li><b style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"><span lang="EN-US">No delays</span></b><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;">: avoid as much as possible work that is sitting between two steps of the development process (what is called WIP – work in process – in the lean jargon). This is true at the process level (the goal is to design a streamlines/ “single piece flow” development organization) and the developer level. A practical goal is to avoid switching tasks as much as possible: focus on one thing and do it right!</span></li><li><b style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"><span lang="EN-US">Quality in</span></b><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"> (right first time) :  this lean principle translates into testing as early as possible (a tenet from agile programming) but also to use all techniques that improve the quality of the code, even at the expense of source code productivity since we all know that it is cheaper not to produce a bug than to remove it later. Here comes, for instance, the practice of pair programming and code reviews, but also “good practices” such as programming guidelines and standards</span></li><li><b style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"><span lang="EN-US">Fast delivery</span></b><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;">: the lean principle is to reduce the “lead time” of the software development process, which requires working on all stages. Removing in-between delays (cf. earlier) is necessary but not enough. <a href="http://en.wikipedia.org/wiki/Continuous_integration">Continuous integration</a> is a core technique to achieve this goal, such as fast deployment techniques.</span></li><li><b style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"><span lang="EN-US">Short deliveries</span></b><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;">: It is more efficient to produce small pieces of software at a high rate than bigger pieces at a lower rate. This is another key principle from lean (“small batches”), which is doubly true for software development: not only are l smaller batches easier to build (a well-known law of software engineering), but the continuous evolution of customers needs and environment makes it easier to adapt with a small batch approach.</span></li><li><b style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"><span lang="EN-US">Less code</span></b><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;">: this is the application of the <a href="http://en.wikipedia.org/wiki/KISS_principle">KISS principle</a> !  Lean software development tries to stay away from complexity (see later in this post).  Unnecessary code may pop up from many sources, lean applies a technique called VSM (Value-Stream Mapping) and a posture of “muda removal”.  Muda (waste) removal means to go through the process with the “eyes of the customer” and remove all that does not produce value from her perspective. VSM is a tool that tracks the value creation and assigns it to each step of the process. Lean software development aims at producing the right product, without unnecessary features. It is also an architecture principle (stay away from complexity) designed at simpler and faster maintenance over the years.</span></li><li><b style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"><span lang="EN-US">Customer participation</span></b><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;">: the best way to produce only what is necessary from the user’s perspective is to ask her frequently ! This is why end-user/customer participation is a tenet of agile programming. When the customer is not available, the principle mixes with “small batches” to become : deliver fast, <a href="http://www.jasonclegg.com/2010/09/speed-of-implementation-how-to-fail-faster-to-succeed-sooner/">fail faster to succeed sooner</a>.</span></li><li><b style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"><span lang="EN-US">Regular work effort</span></b><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"> : the leveling of the effort is a key principle from extreme programming, the equivalent of “heijunka” from lean programming. A few years ago, when I was still a CIO, I started thinking about “<b>extreme IT</b>” (applying extreme programming at the information system level), and the <a href="http://informationsystemsbiology.blogspot.com/2009/01/soa-is-much-too-young-to-be-dead.html">sustainability of the effort</a> is a crucial point. Regularity has a counterpart, which is discipline. Using methods and tools (such as configuration management, source code versioning, automated testing) is crucial. One should take a look at the <a href="http://www.lean-it-summit.com/">wonderful talk from Mark Striebeck</a> on “Creating a testing culture” at Google.</span></li><li><b style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"><span lang="EN-US">Minimize handovers</span></b><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;">: complex tasks, such as writing software, are better accomplished when the responsibility chain is not truncated into too many segments. Working as a team is the only alternative to deliver complex projects (this is the topic of <a href="http://organisationarchitecture.blogspot.com/2011/05/processus-et-entreprise-20-est.html">my third book</a>). This is another insight from the <a href="http://agilemanifesto.org/">Agile Manifesto</a>, this is why today’s best practice it to assemble multi-disciplinary teams including software developers, marketers and designers.</span></li></ul><span style="font-family: 'Trebuchet MS', sans-serif;">There are a number of interesting books on this topic. The most classical ones are those from </span><a href="http://www.poppendieck.com/" style="font-family: 'Trebuchet MS', sans-serif;">Mary Poppendieck</a><span style="font-family: 'Trebuchet MS', sans-serif;">, such as “Implementing Lean Software Development: from Concept to Cash”.  I had the pleasure of presenting my last book at the </span><a href="http://www.lean-it-summit.com/" style="font-family: 'Trebuchet MS', sans-serif;">Lean IT Summit</a><span style="font-family: 'Trebuchet MS', sans-serif;"> a month ago and to meet with Mary Poppendieck (a number of great talks, including hers, Michael Ballé’s and Georges Striebeck’s, are now available </span><a href="http://www.lean-it-summit.com/" style="font-family: 'Trebuchet MS', sans-serif;">online</a><span style="font-family: 'Trebuchet MS', sans-serif;">).</span><br /><div class="MsoNormal"><span lang="EN-US"><span style="font-family: 'Trebuchet MS', sans-serif;">There are also books which are not about software development <i>per se</i> but very closely related. I am especially fond of two books which <a href="http://organisationarchitecture.blogspot.com/2011/08/lean-startup-et-lean-product.html">I have reviewed in my French blog</a>:<o:p></o:p></span></span></div><div class="MsoNormal"></div><ul style="text-align: left;"><li><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"><a href="http://theleanstartup.com/">The Lean Startup</a>, from Eric Riess,</span></li><li><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"><a href="http://www.startuplessonslearned.com/2009/07/principles-of-product-development-flow.html">The principles of Product Development Flow</a>, from Donald G. Reinertsen</span></li></ul><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-align: justify;">Thanks to <a href="http://devops.fr/">Guillaume Fortaine</a>, I have come to learn about <a href="http://en.wikipedia.org/wiki/DevOps">Devops</a>. Quoting from Wikipedia, “</span><b style="font-family: 'Trebuchet MS', sans-serif; text-align: justify;"><span lang="EN-US" style="background-attachment: initial; background-clip: initial; background-color: white; background-image: initial; background-origin: initial;">DevOps</span></b><span lang="EN-US" style="background-attachment: initial; background-clip: initial; background-color: white; background-image: initial; background-origin: initial; font-family: 'Trebuchet MS', sans-serif; text-align: justify;"><span style="text-align: -webkit-auto;">" is an emerging set of principles, methods and practices for communication, collaboration and integration between<span class="apple-converted-space"> </span></span></span><a href="http://en.wikipedia.org/wiki/Software_development" style="background-attachment: initial; background-clip: initial; background-origin: initial; font-family: 'Trebuchet MS', sans-serif; text-align: -webkit-auto;" title="Software development"><span lang="EN-US" style="background-attachment: initial; background-clip: initial; background-color: white; background-image: initial; background-origin: initial; color: #0b0080; text-decoration: none;">software development</span></a><span class="apple-converted-space" style="font-family: 'Trebuchet MS', sans-serif; text-align: justify;"><span lang="EN-US" style="background-attachment: initial; background-clip: initial; background-color: white; background-image: initial; background-origin: initial;"><span style="text-align: -webkit-auto;"> </span></span><span lang="EN-US" style="background-attachment: initial; background-clip: initial; background-color: white; background-image: initial; background-origin: initial;">(application/software engineering) and<span class="apple-converted-space"> </span></span></span><a href="http://en.wikipedia.org/wiki/Web_operations" style="background-attachment: initial; background-clip: initial; background-origin: initial; font-family: 'Trebuchet MS', sans-serif; text-align: -webkit-auto;" title="Web operations"><span lang="EN-US" style="background-attachment: initial; background-clip: initial; background-color: white; background-image: initial; background-origin: initial; color: #0b0080; text-decoration: none;">IT operations</span></a><span class="apple-converted-space" style="font-family: 'Trebuchet MS', sans-serif; text-align: justify;"><span lang="EN-US" style="background-attachment: initial; background-clip: initial; background-color: white; background-image: initial; background-origin: initial;"><span style="text-align: -webkit-auto;"> </span></span><span lang="EN-US" style="background-attachment: initial; background-clip: initial; background-color: white; background-image: initial; background-origin: initial;">(systems administration/infrastructure) professionals</span>”</span><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-align: justify;">.  The more I read about Devops (<a href="http://devops.com/">Devops.com</a> is a treasure trove of great articles), the more I find that Devops is the missing link between Agile/Extreme Programming and lean management (Toyota-style, hence Lean-Startup-style).  Although Devops claims to “help finishing what Agile Development started”, there are different flavors in what one may read on the web: <a href="http://www.jedi.be/blog/2010/02/12/what-is-this-devops-thing-anyway/">bridging the development /admin & operations gap</a>, <a href="http://blog.octo.com/devops-ou-le-lean-applique-aux-activites-it-du-developpement-a-la-production/">making lean IT happen</a>, <a href="http://devops.com/2011/10/22/overcoming-organizational-hurdles/">making collaboration happen</a> (and <a href="http://dyn.com/allops-evolution-from-the-devops-culture-movement/">also</a>),  <a href="http://www.qualitystreet.fr/2011/09/30/devops-cest-le-top/">bringing agility to operations</a>,  which in turns makes it easier to leverage the benefits of new computing resources such as <a href="http://devops.com/2011/10/23/devops-in-the-cloud-explained/">cloud programming</a>. My first obvious interest with Devops has been the practical deployment of lean IT, as a followup of what I just explained earlier. For instance, Devops promote a number of interesting tools related to deployment automation, such  as <a href="http://devops.com/2011/07/09/glu-deployment-automation-video/">GLU</a>. It turns out that the “Cloud and Devops intersection” is also quite promising. Indeed, leveraging <a href="http://informationsystemsbiology.blogspot.com/2008/11/very-cloudy-weather-will-it-rain-on.html">the strengths of cloud programming</a> requires a shift in development/architecture culture that requires a “Devops-like” approach.</span><br /><div class="MsoNormal" style="text-align: justify;"><span style="font-family: 'Trebuchet MS', sans-serif;"><span lang="EN-US"><br /></span></span></div><div class="MsoNormal"><span lang="EN-US"><span style="font-family: 'Trebuchet MS', sans-serif;">It happens that I gave a talk last week at the <a href="http://www.mdday.fr/">Model-Driven Day</a> about « Complexity, Modularity and Abstraction” (the talk is available in the “My Shared Document” box on the left). The talk is about, among other things:<o:p></o:p></span></span></div><div class="MsoNormal"></div><ul style="text-align: left;"><li><b style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"><span lang="EN-US">Complexity </span></b><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;">(why it is important, how to measure it and to <a href="http://informationsystemsbiology.blogspot.com/2010/01/taming-information-systems-complexity.html">tame complexity</a> – since avoiding complexity altogether is not an option at the IS level)</span></li><li><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"><span style="font: normal normal normal 7pt/normal 'Times New Roman';"> </span></span><b style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"><span lang="EN-US">Sustainability</span></b><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;">: how to transform enterprise architecture into a regular practice. This is related to the concept of “<b>extreme IT</b>” : avoid the “heroic struggle” to move towards the “continuous transformation of the information system”. This is a major reason why I have been advocating for <a href="http://informationsystemsbiology.blogspot.com/2009/05/soa-tale-of-two-cities.html">SOA</a>as a company-wide <a href="http://capirossi.org/">enterprise architecture</a>practice for many years.</span></li><li><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;">“<b>Architecture-oriented services</b>” : I made this pun to emphasize the difficulty to produce the “right” services through SOA. “Architecture-oriented” means services that have the right level of abstraction, that are modular and “easy-to-compose”. To my knowledge, there is no easy recipe for this, but the wisdom and folklore of 40 years of software architecture design apply.</span></li><li><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"><span style="font: normal normal normal 7pt/normal 'Times New Roman';"> </span></span><b style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;"><span lang="EN-US">Cloud computing</span></b><span lang="EN-US" style="font-family: 'Trebuchet MS', sans-serif; text-indent: -18pt;">: I have added a small addition about “cloud-ready architecture” in the 4th edition of <a href="http://www.amazon.fr/Urbanisation-SOA-BPM-%C3%A9dition-point/dp/2100566369/ref=sr_1_1?ie=UTF8&qid=1322377931&sr=8-1">my first book</a>. I strongly believe that Information Systems will change in a spectacular manner when we learn how to exploit massively parallel architecture (using tools/approaches such as Hadoop/MapReduce). Using Cloud Computing to provide with a few Saas-based front-office services is nice and relevant, but the big change comes when cloud-computing is applied to back-office services (provisioning, billing, data mining). This requires an architecture change, but mostly requires a culture change.</span></li></ul><span style="font-family: 'Trebuchet MS', sans-serif;">The thought that prompted me to write this post is as follows. Devops is the missing link between the themes of my MDD talk: managing complexity, delivering agility/modularity and moving to the new century of massively parallel computing (including, but not restricted to, cloud computing). I speak of "Cloud Programming" in the title because I agree with a comment made by <a href="http://devops.com/2011/10/23/devops-in-the-cloud-explained/">Georges Reese</a>: the Cloud is a computing resources defined by its API, i.e., a resource that is managed through programming. The exact quote is:</span><br /><div style="text-align: left;"><span style="text-align: justify;"><i style="color: #333333; font-family: 'Lucida Grande', Verdana, Arial, sans-serif; line-height: 19px;">Cloud is, for the purposes of this discussion, is sort of the pinnacle of SOA in that it makes everything controllable through an API. If it has no API it is not Cloud. If you buy into this then you agree that everything that is Cloud is ultimately programmable</i>.</span></div><span style="font-family: 'Trebuchet MS', sans-serif;"> Not only the cloud is more agile (up/down scalability), it may be controled by the piece of software which is using it. From a systemic perspective, it's a "whole new game" :)</span></div>

Lean IT, Devops and Cloud Programming

I have become more and more interested with lean IT over the years. It started with the book "The Art of Lean Software Development...

Read more »