Lean, Scrum, Agile and Extreme Programming

<div dir="ltr" style="text-align: left;" trbidi="on"><br /><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">Two thousand years ago, Epictetus told us “<b>i</b></span><b><span lang="EN-US" style="background: white; color: #222222; mso-ansi-language: EN-US; mso-bidi-font-family: Arial;">t is impossible for a<span class="apple-converted-space"> </span></span></b><em><b><span lang="EN-US" style="background-color: white;">man</span></b></em><span class="apple-converted-space"><b><span lang="EN-US" style="background: white; color: #222222; mso-ansi-language: EN-US; mso-bidi-font-family: Arial;"> </span></b></span><b><span lang="EN-US" style="background: white; color: #222222; mso-ansi-language: EN-US; mso-bidi-font-family: Arial;">to<span class="apple-converted-space"> </span></span></b><em><b><span lang="EN-US" style="background-color: white;">learn what he thinks he already knows</span></b></em><span lang="EN-US">”. This principle is still one of the main reasons why it is so difficult to deploy change, when new (development) methods are concerned. I have been advocating for lean and agile methods for many years and the most common pattern I hear is: “<i>I know, I do it already</i>” :)</span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">The goal of this post is to focus on the difference between agile in general, SCRUM in particular, and lean software development. I have been speaking about lean software development for three years and I have found that the (legitimate) confusion between lean and agile does not help. I keep hearing “<i>I already know about agile, scrum … or lean</i>”. I can relate to this. I started as a research scientist in the field of programming languages and software engineering twenty-five years ago, and was there (in the US & the academic community) when agile was born, I knew personally some the founders of the </span><a href="http://agilemanifesto.org/principles.html"><span lang="EN-US">Agile Manifesto</span></a><span lang="EN-US">. I started practicing agile and extreme programming when I returned to France to create the </span><a href="http://e-lab.bouygues.com/"><span lang="EN-US">e-Lab</span></a><span lang="EN-US">. I brewed my own agile philosophy for a small team of “PhD programmers” under the designation of </span><a href="http://organisationarchitecture.blogspot.fr/2011/12/grammaire-de-linaction-et-programmation.html"><span lang="EN-US">Fractal Programming</span></a><span lang="EN-US">. Since then, I have joined larger organizations and tried to promote agile methods. At first, I was promoting what I once knew and tried, but I have quickly discovered that the agile movement is growing constantly, and that we know much more today than what you learned yesterday. Hence the Epitectus quotation.<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US"><br /></span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">The topic of mapping the similarities and differences between lean development and agile is quite interesting because the influences and relationships make for an intricate pattern. Agile methods and its various subsequent developments have been implementing “lean principles” for many years. Lean Software Development, on the other hand, inherits from the knowledge built by agile communities. There is already a number of interesting post on this topic. </span><a href="http://www.agileweboperations.com/lean-agile-devops-related"><span lang="EN-US">Matthias Marschall’s contribution</span></a><span lang="EN-US"> is a great synthesis which is very similar to what I’ll write in a few minutes. He shows the strong relationships and the differences, and point out at the development process dimension: <i>lean</i> helps to get a larger perspective. His </span><a href="http://www.agileweboperations.com/lean-agile-devops-related"><span lang="EN-US">slideshare</span></a><span lang="EN-US"> is also interesting, but is not structured enough from my “conceptual” perspective, which I why I found useful to write my own analysis down. Another </span><a href="http://thehypertextual.com/2012/10/16/agile-et-hyperlocal-atbdx/"><span lang="EN-US">great contribution by Cecil Dijoux</span></a><span lang="EN-US"> points out the influences but stresses the benefits of lean as providing a larger perspective, especially </span><a href="http://thehypertextual.com/2012/04/25/lean-as-the-solution-to-scale-agile-methodologies/"><span lang="EN-US">the ability to scale up agile methods</span></a><span lang="EN-US"> to enterprise management and transformation. Cecil talks about </span><a href="http://www.morisseauconsulting.com/index.php/livre-kanban-pour-lit/"><span lang="EN-US">Kanban</span></a><span lang="EN-US">, to emphasize that it goes further than the traditional visual management practices of SCRUM. There are so many other examples. For instance, talks about agile methods such as </span><a href="http://fr.slideshare.net/LaurentSarrazin/2012-0525agile-francesweetrupture03final"><span lang="EN-US">Laurent Sarrazin</span></a><span lang="EN-US">’s which makes the influence of lean principles obvious. Laurent’s is a great example of a modern talk about agile, which goes much deeper that what I was hearing 15 years ago. In a reciprocate manner, great pieces about lean development, such as </span><a href="http://welovelean.com/write-up-10-things-ive-learnt-about-lean-star"><span lang="EN-US">Welovelean</span></a><span lang="EN-US">’s, are still very close to the founding principles of the Agile Manifesto.<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US"><br /></span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">The claim of this post is : <b>there is more to lean software development than a great synthesis of the best agile methods</b>. To make my point, I will briefly summarize the key ideas of the successive steps of the evolution path from </span><a href="http://en.wikipedia.org/wiki/Agile_software_development"><span lang="EN-US">Agile</span></a><span lang="EN-US">, </span><a href="http://en.wikipedia.org/wiki/Extreme_programming"><span lang="EN-US">XP</span></a><span lang="EN-US">, </span><a href="http://en.wikipedia.org/wiki/Scrum_(development)"><span lang="EN-US">SCRUM</span></a><span lang="EN-US"> to </span><a href="http://en.wikipedia.org/wiki/Lean_software_development"><span lang="EN-US">Lean Software Development</span></a><span lang="EN-US">. I need to stress out that the following description is both too brief and incomplete. It is quite succinct because (a) I have already talked about this in </span><a href="http://informationsystemsbiology.blogspot.fr/2011/11/lean-it-devops-and-cloud-programming.html"><span lang="EN-US">previous posts</span></a><span lang="EN-US"> (b) it is easy to find better references for Agile (</span><a href="http://agilemanifesto.org/principles.html"><span lang="EN-US">manifesto</span></a><span lang="EN-US">), </span><a href="http://www.extremeprogramming.org/"><span lang="EN-US">XP</span></a><span lang="EN-US">, </span><a href="http://www.scrum.org/Resources/What-is-Scrum"><span lang="EN-US">SCRUM</span></a><span lang="EN-US"> or </span><a href="http://fr.slideshare.net/jpvajda/lean-software-development-principles"><span lang="EN-US">Lean Software</span></a><span lang="EN-US">. It is incomplete by construction, since to promote clarity, I have only selected three additional ideas/principles when moving from Agile to XP, then to SCRUM, then to LSF.<o:p></o:p></span></span></div><div class="MsoNormal"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">Let me then characterize the essence of agile (when it started) as the combination of: <o:p></o:p></span></span></div><div class="MsoNormal"></div><ol style="text-align: left;"><li><a href="http://blog.extremeplanner.com/2006/08/small-teams-make-better-software.html" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Small teams</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"> : a strong emphasis on strong interactions.</span></li><li><a href="http://fr.slideshare.net/SteveRogalsky/agile101-small-batches" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Small batches</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"> : break a larger project into manageable chunks (sprints), do them one at a time and constantly re-evaluate what the next chunk will be.</span></li><li><a href="http://agile.dzone.com/articles/qa-agile-approach-0" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Time boxing</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"> : there is nothing like a time constraint to help you focus on what really matters. Deliver less, but deliver on time.</span></li><li><a href="http://www.agilemodeling.com/essays/executableSpecifications.htm" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Coevolution of code/design/architecture</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"> : the end of the waterfall model, the code and its <a href="http://www.leanessays.com/2011/08/dont-separate-design-from.html">requirements</a>are built continuously.</span></li><li><a href="http://heratech.wordpress.com/2009/12/28/agile-tenet-6%E2%80%93-face-to-face-communication/" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Role of face-to-face communication</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"> : A key insight of agile is that to break the waterfall model, you need to reintroduce face-to-face interaction and banish the email push, which breaks the responsibility and engagement chains.</span></li><li><a href="http://www.agilemodeling.com/artifacts/userStory.htm" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">User stories</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"> : the only way to capture a need is to give it some flesh and make it into a story. Read Daniel Pink’s “</span><a href="http://en.wikipedia.org/wiki/A_Whole_New_Mind" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">A Whole New Mind</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">” to understand why.</span></li></ol><br /> <div class="MsoListParagraphCxSpLast" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;">There is a lot of </span><i style="font-family: Verdana, sans-serif;">lean</i><span style="font-family: Verdana, sans-serif;"> in these principles & practices. Obviously once cannot reduce agility to this short list of principles, I should also explain a lot of associated practices (supported with tools) regarding the development process. Let me now turn to Extreme Programming, which I will characterize with three additional ideas (or increased emphasis):</span></div><div class="MsoListParagraphCxSpLast" style="text-align: justify;"></div><ol><li><a href="http://en.wikipedia.org/wiki/Test-driven_development" style="font-family: Verdana, sans-serif; text-align: left; text-indent: -18pt;"><span lang="EN-US">Test-driven development</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-align: left; text-indent: -18pt;">: the best way to develop code is to write test cases first.</span></li><li><a href="http://www.extremeprogramming.org/rules/overtime.html" style="font-family: Verdana, sans-serif; text-align: left; text-indent: -18pt;"><span lang="EN-US">Sustainable pace</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-align: left; text-indent: -18pt;">: the only way to satisfy a customer in the software world is to last, because software is a “living thing” that evolves constantly. Hence, the rhythm must be set to reach long-term goals and respect everyone in the development process.</span></li><li><a href="http://elegantcode.com/2011/01/21/book-review-extreme-programming-explained/" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Code is valuable</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">: XP was the first method to significantly raise the status of code, through practices such as code reviews, refactoring or pair programming and principles such as “elegant code”, “do it right the first time”, etc.</span></li></ol><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">Here, as well as in the following section on SCRUM, lean is everywhere </span><span lang="EN-US">J</span><span lang="EN-US"> That is, we can see these principles as the application of lean principles to software development, derived from the <a href="http://en.wikipedia.org/wiki/Toyota_Production_System">TPS</a>. I would then characterize SCRUM with the following additional contributions:<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"></div><ol><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-align: left; text-indent: -18pt;"><span style="font-size: 7pt;"> </span></span><a href="http://www.xqa.com.ar/visualmanagement/" style="font-family: Verdana, sans-serif; text-align: left; text-indent: -18pt;"><span lang="EN-US">Visual Management</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-align: left; text-indent: -18pt;">: give the team a project room, use the walls to share release notes, the backlog, planning information, etc.</span></li><li><a href="http://en.wikipedia.org/wiki/Stand-up_meeting" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Practices and Rites</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">: SCRUM is practical method that makes agile easier to learn through its rites, such as standup meetings.</span></li><li><a href="http://zacgery.blogspot.fr/2012/11/reflections-in-scrums-silver-mirror.html" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Reflection</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">: SCRUM proposes </span><a href="http://www.mountaingoatsoftware.com/scrum/sprint-retrospective/" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">sprint  retrospectives</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">, a great tool to promote continuous improvement. One could say that standing back is part of agile, but SCRUM brings a true emphasis on “doing it for real”.</span></li></ol><div class="MsoNormal"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">These twelve sets of practices make a maturity model for SCRUM. To qualify to lean software development, I would add another three steps:<o:p></o:p></span></span></div><div class="MsoNormal"></div><ol style="text-align: left;"><li><a href="http://en.wikipedia.org/wiki/Kanban_(development)" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">Kanban</span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">: the application of visual management to the visualization of process flow and work in process (WIP). Kanban is a critical tool to reduce WIP and to implement pull (just in time), two key lean principles.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><a href="http://en.wikipedia.org/wiki/Kaizen">Kaizen</a> :  the heart of the lean approach towards problem solving. Lean makes problem solving a team activity and a learning tool. There are many associated tools and practices such as the “<a href="http://en.wikipedia.org/wiki/5_Whys">Five whys</a>” or <a href="http://sloanreview.mit.edu/the-magazine/2009-summer/50408/toyotas-secret-the-a3-report/">Toyota’s A3</a>.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;"><a href="http://en.wikipedia.org/wiki/5S_(methodology)">5S</a> and <a href="http://www.allaboutagile.com/lean-principles-1-eliminate-waste/">waste removal</a> : sort, clean-up, organize, focus on simplicity and writing less code. Some of it is part of XP, <a href="http://leansoftwarepractices.com/">but lean goes further</a>, with more tools and deeper practices (e.g. <a href="http://en.wikipedia.org/wiki/Value_stream_mapping">value-stream mapping</a>).</span></li></ol><br /> <div class="MsoListParagraphCxSpMiddle" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;">This is a simplified view of lean software (although the combined practices of the 15 previous bullet points represents a fair amount of commitment), see the seven principles of </span><a href="http://www.poppendieck.com/" style="font-family: Verdana, sans-serif;">Mary and Tom Poppendieck</a><span style="font-family: Verdana, sans-serif;"> or my own description of </span><a href="http://informationsystemsbiology.blogspot.fr/2012/07/lean-software-factory.html" style="font-family: Verdana, sans-serif;">Lean Software Factory</a><span style="font-family: Verdana, sans-serif;"> for more details. A great practical introduction to lean software development is “</span><a href="http://www.crisp.se/file-uploads/Lean-from-the-trenches.pdf" style="font-family: Verdana, sans-serif;">Lean from the Trenches</a><span style="font-family: Verdana, sans-serif;">” by Henrik </span>Kniberg<span style="font-family: Verdana, sans-serif;">.</span></div><div class="MsoListParagraphCxSpMiddle" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><br /></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><b><span lang="EN-US">Another key evolution of the past 20 years is the scope of agile/lean development method, which has grown from software development to product innovation</span></b><span lang="EN-US">. In the spirit of keeping this post simple, I will consider the scope with four successively embedded steps (from the smallest to the largest):<o:p></o:p></span></span></div><div class="MsoNormal"></div><ul style="text-align: left;"><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">Software development is the original scope of agile methods. It obviously includes testing, as well as the proper code management & configuration tools to make coding as efficient as possible. See my reference to Curt Hibbs in my <a href="http://informationsystemsbiology.blogspot.fr/2012/07/lean-software-factory.html">previous post on software factories</a>.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">The next step is to move to <a href="http://en.wikipedia.org/wiki/Continuous_integration">continuous integration</a>. The ability to generate a “fresh product” every night (for instance) that can be fully tested, with as many automated test as possible, yields a completely different culture and level of innovation. This requires a significant amount of tools and training. This is the case where you need to change the tools in order to change the culture.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">The third step is <a href="http://radar.oreilly.com/2009/03/continuous-deployment-5-eas.html">continuous deployment</a>, which is best illustrated by <a href="http://en.wikipedia.org/wiki/DevOps">Devops</a>. I have mentioned Devops <a href="http://informationsystemsbiology.blogspot.fr/2011/11/lean-it-devops-and-cloud-programming.html">already</a>, the big step is to include operations in the lean/agile process. Moving from CI to CD requires more tools, more automation … and a big culture change since it require to move from “agile in the lab” to “agile in the field”.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">The last step is <a href="http://asqcincinnati.org/Library/continuousimprovement/ImprovementToInnovation.pdf">continuous product improvement and innovation</a>, which is beautifully explained in <a href="http://organisationarchitecture.blogspot.fr/2011/08/lean-startup-et-lean-product.html">The Lean Startup</a> from Eric Ries. In addition to continuous deployment, we need to gather continuous feedback (e.g., with <a href="http://en.wikipedia.org/wiki/A/B_testing">A/B testing</a>) and re-inject this knowledge into the development cycle. A great source to better understand lean startup principles is <a href="http://www.ashmaurya.com/">Ash Maurya’s blog</a>, and these principles do not apply to startups only!</span></li></ul><br /> <div class="MsoNormal" style="text-align: justify;"><b><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">Lean Software Development is the combination of both dimensions: the depth of lean practices and the scope</span></span></b><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">. I could easily derive a maturity index from this: the maturity level would be the product of:  (a) how many of the 15 principles are really in action, times (b) what is the scope of the lean/agile process. I won’t go into this direction because lean software is much more than a development method; it is a work philosophy, in the “<a href="http://en.wikipedia.org/wiki/The_Toyota_Way">Toyota Way</a>” sense. It is mostly a work philosophy that is built on the <a href="http://organisationarchitecture.blogspot.fr/2012/11/motivation-lean-et-stress.html">motivation and satisfaction</a> of the software producers. A key insight from “<a href="http://www.amazon.fr/Implementing-Lean-Software-Development-Concept/dp/0321437381">Implementing Lean Software Development</a>” is that one must move away from a project-oriented culture based on contracts towards a product-oriented culture based on collaboration. A wonderful illustration of the scope of the lean transformation is the great book “<a href="http://www.amazon.fr/Les-G%C3%A9ants-Web-Pratiques-Architecture/dp/2952589534/ref=sr_1_1?ie=UTF8&qid=1354347141&sr=8-1">Les Géants du Web</a>”, just released by <a href="http://blog.octo.com/tag/les-geants-du-web/">Octo</a>. Its subtitle is “culture / practice / architecture”; most of what I have just described is beautifully explained and illustrated. There is lean and agile everywhere in that book, but this is a book for everyone, not reserved to managers or software developers. The book is about what makes the best “Web companies” successful, from a culture, practice and architecture point of view. However, the similarity with the lean philosophy is striking.</span><o:p></o:p></span></div></div>

Lean, Scrum, Agile and Extreme Programming

Two thousand years ago, Epictetus told us “ i t is impossible for a   man   to   learn what he thinks he already knows ”. This principle is ...

Read more »