Big Data hides more than one paradigm shift

<div dir="ltr" style="text-align: left;" trbidi="on"><div class="MsoNormal"><br /></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">This post originates from a report which I wrote this summer for the </span><a href="http://www.euro-case.org/index.php/academy/item/72-national-academy-of-technologies-of-france"><span lang="EN-US">NATF</span></a><span lang="EN-US">, which proposes a summary of what the ICT commission learned from its two-years cycle of interviews about Big Data. The commission decided to investigate about the impact of Big Data on French economy in 2012. Big Data is such a popular and hyped topic that it was not clear, at first, if a report would be necessary. So many books and reports have been published in the past two years (see the extract from the bibliography in the next section) that it made little sense to add a new one. However, throughout our interviews, and thanks to the </span><a href="http://www.naefrontiers.org/Symposia/EU-USFOE/22791/32009.aspx"><span lang="EN-US">FoE conference</span></a><span lang="EN-US"> that NATF co-organized last year with NAE in Chantilly – which included Big Data as one of its four topics – we came to think that there was more to say about the topic than the usual piece about new customer insights, data scientists, the internet of things and the opportunities for new services.<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US"><br /></span></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">Today I will focus on two ideas that may be characterized as paradigm shifts. I will keep this post short so it may be seen as a “teaser” for the full report which should be available soon. The first paradigm shift is a new way to analyze data based on systemic cycles and the real-time analysis of correlation. The old adage “correlation is not causation” is made “obsolete” because data mining is embedded into an operational loop that is judged, not by the amount of knowledge that is extracted, but by the dollar amount of new business that is generated. The second paradigm shift is about programming: Big data entails a new way to produce code in a massively distributed environment. This disruption comes from two fronts: the massive volume of data requires to distribute both data and procedure, on the one hand, and algorithmic tuning needs to be automated, on the other hand. Algorithms are grown as much as they are designed, they are derived from data though machine learning.<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">The full report contains an overview of what Big Data is because a report from NATF needs to be self-contained, but this is not necessary for a blog post. I assume that the reader has some familiarity with the Big Data topic. Otherwise, the </span><a href="http://en.wikipedia.org/wiki/Big_data"><span lang="EN-US">Wikipedia page</span></a><span lang="EN-US">is a good place to start, followed by the upcoming bibliography entries, first of which Viktor Mayer-Schönberger and Kenneth Cukier’s book.<o:p></o:p></span></span></div><div class="MsoNormal"><br /></div><div class="MsoNormal" style="text-indent: 0px;"><span style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US"><span style="font-size: large;">1. <b>Big Data – A Revolution That Will Transform How We Live</b></span></span></span></div><div class="MsoNormal"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRIt_2eZnECBUOcv2krHtoiJHNNB9p6hn9Jff6G6arlKJvj2AkvJ7ZdYZRjLXyYGcd5Q2p8LYjE8pgnS6bGd30ZnMBILKS57-Owe98B9YCbf7DpbrgB_i1P2XrDXe-Ygig2SU61TFng_g/s1600/BigData.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRIt_2eZnECBUOcv2krHtoiJHNNB9p6hn9Jff6G6arlKJvj2AkvJ7ZdYZRjLXyYGcd5Q2p8LYjE8pgnS6bGd30ZnMBILKS57-Owe98B9YCbf7DpbrgB_i1P2XrDXe-Ygig2SU61TFng_g/s1600/BigData.jpg" /></a></div><div class="MsoNormal"><br /></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">This is the title of the book from Viktor Mayer-Schönberger and Kenneth Cukier, which covers the most famous paradigm shift of Big Data, which is its ability to transform our lives, from hard science, such as medicine, to marketing. The paradigm shift comes from the combination of what technology makes possible today – the ability to analyze very large amount of heterogeneous data in a very short amount of time – and the availability of the relevant data, which are the traces of our lives that have become digital. Thanks to the web, to smartphones, to technology which is everywhere in our lives and objects, there is a continuous stream of information that describes the world and our actions. Big Data may be described as the Information Technology which is able to mine this “digital logs” and produce new insights, opportunities and services. The constant improvement of technology (from </span><a href="http://en.wikipedia.org/wiki/Moore's_law"><span lang="EN-US">Moore’s Law</span></a><span lang="EN-US"> about processing to </span><a href="http://en.wikipedia.org/wiki/Mark_Kryder"><span lang="EN-US">Kryder’s Law</span></a><span lang="EN-US"> about storage) is matched by the increase in digital details about our lives. New connected objects, sensors and the growth of IoT (<a href="http://en.wikipedia.org/wiki/Internet_of_Things">Internet of Things</a>) mean that we are only seeing the beginning of what Big Data will be able to do in the future.  <o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">One of the reason for not discussing these themes further is that the book from Viktor Mayer-Schönberger and Kenneth Cukier covers them very well so that I encourage you to read it. The other reason is that there are many other sources that develop these theses. Here is a short extract from our report’s bibliography :<o:p></o:p></span></span></div><div class="MsoListParagraph" style="margin: 0cm 0cm 6pt; text-indent: 0.55pt;"><span style="background-color: white; font-family: Verdana, sans-serif; font-size: x-small;"><span lang="EN-US">[1] Commission Anne Lauvergeon. </span><a href="http://innovation-2030.entreprises.gouv.fr/pdf/Rapport_Innovation_BDV4.pdf"><i>Un principe et sept ambitions pour l’innovation</i></a>. 2013.</span><br /><span style="background-color: white; font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">[2] John Podesta & al. </span><a href="http://www.whitehouse.gov/sites/default/files/docs/big_data_privacy_report_may_1_2014.pdf" style="font-family: Verdana, sans-serif; font-size: small; text-indent: 0.55pt;"><i><span lang="EN-US">Big Data : Seizing Opportunities, preserving values</span></i></a><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">. Executive Office of the President, May 2014.</span><br /><span style="background-color: white; font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">[3] François Bourdoncle. </span><i style="font-family: Verdana, sans-serif; font-size: small; text-indent: 0.55pt;">“</i><a href="http://ecole.org/fr/journal/J124" style="font-family: Verdana, sans-serif; font-size: small; text-indent: 0.55pt;"><i>Peut-on créer un écosystème français du Big Data ?</i></a><i style="font-family: Verdana, sans-serif; font-size: small; text-indent: 0.55pt;">”</i><span style="background-color: white; font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">, Le Journal de l’Ecole de Paris n°108, Juillet/Aout 2014.</span><br /><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">[5] Viktor Mayer-Schönberger, Kenneth Cukier. </span><a href="https://www.youtube.com/watch?v=bYS_4CWu3y8" style="font-family: Verdana, sans-serif; font-size: small; text-indent: 0.55pt;"><i><span lang="EN-US">Big Data – A Revolution That Will Transform How We Live, Work and Think</span></i></a><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">. </span><span style="background-color: white; font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">John Murray, 2013.</span><br /><span style="background-color: white; font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">[8] Gilles Babinet. </span><a href="http://organisationarchitecture.blogspot.fr/2014/03/entreprises-numeriques-et-homeostasie.html" style="font-family: Verdana, sans-serif; font-size: small; text-indent: 0.55pt;"><i>L’ère numérique, un nouvel âge de l’humanité : Cinq mutations qui vont bouleverser votre vie</i></a><span style="background-color: white; font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">. </span><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">Le Passeur, 2014.</span><br /><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">[10] Phil Simon. </span><a href="http://readwrite.com/2011/11/23/the_age_of_the_platform_book_excerpt" style="font-family: Verdana, sans-serif; font-size: small; text-indent: 0.55pt;"><i><span lang="EN-US">The Age of The Platform – How Amazon, Apple, Facebook and Google have redefined business</span></i></a><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">. Motion Publishing, 2011.</span><br /><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">[12] IBM Global Business Services, « </span><a href="http://www-935.ibm.com/services/multimedia/Anaytics.pdf" style="font-family: Verdana, sans-serif; font-size: small; text-indent: 0.55pt;"><i><span lang="EN-US">Analytics : Real-world use of big data in telecommunications – How innovative communication service providers are extracting value from uncertain data</span></i></a><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">”. IBM Institute for Business Value, Avril 2013.</span><br /><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">[13] Thomas Dapp. “</span><a href="http://www.dbresearch.com/PROD/DBR_INTERNET_EN-PROD/PROD0000000000334340/Big+data+-+the+untamed+force.pdf" style="font-family: Verdana, sans-serif; font-size: small; text-indent: 0.55pt;"><i><span lang="EN-US">Big Data – The untamed force</span></i></a><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">”, Deutsche Bank Research, May 5, 2014.</span><br /><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">[15] David Lazer, Ryan Kennedy, Gary King, Alessandro Vespignani. “</span><a href="http://blogs.iq.harvard.edu/netgov/The%20Parable%20of%20Google%20Flu%20%28WP-Final%29.pdf" style="font-family: Verdana, sans-serif; font-size: small; text-indent: 0.55pt;"><i><span lang="EN-US">The Parable of Google Flu: Traps in Big Data Analysis</span></i></a><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small; text-indent: 0.55pt;">”</span><br /><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small;">[16] Tim Harford. “</span><a href="http://www.ft.com/intl/cms/s/2/21a6e7d8-b479-11e3-a09a-00144feabdc0.html#axzz3GYmvXT5G" style="font-family: Verdana, sans-serif; font-size: small;"><i><span lang="EN">Big data: are we making a big mistake</span></i><span lang="EN">?</span></a><span lang="EN" style="font-family: Verdana, sans-serif; font-size: x-small;">”, Financial Times, </span><span lang="EN" style="font-family: Verdana, sans-serif; font-size: x-small;"> </span><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small;">March 28^th, 2014. </span><br /><span style="background-color: white; font-family: Verdana, sans-serif; font-size: x-small;">[19] Octo Technology. </span><a href="http://www.slideshare.net/OCTOTechnology/octo-2012-les-pratiques-des-gants-du-web" style="font-family: Verdana, sans-serif; font-size: small;"><i>Les géants du Web : Culture – Pratiques - Architecture</i></a><span style="background-color: white; font-family: Verdana, sans-serif; font-size: x-small;">. </span><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small;">Octo 2012.</span><br /><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small;">[21] Tony Hey, Stewart Tansley, Kristin Tolle (eds). </span><a href="http://research.microsoft.com/en-us/collaboration/fourthparadigm/4th_paradigm_book_complete_lr.pdf" style="font-family: Verdana, sans-serif; font-size: small;"><i><span lang="EN-US">The Fourth Paradigm – Data-Intensive Scientific Discovery</span></i></a><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small;">. Microsoft Research, 2009.</span><br /><span style="font-family: Verdana, sans-serif; font-size: x-small;"><span lang="EN-US">[22] Max Lin. “</span><a href="http://fr.slideshare.net/npinto/harvard-cs264-09-machine-learning-on-big-data-lessons-learned-from-google-projects-max-lin-google-research"><i><span lang="EN-US">Machine Learning on Big Data – Lessons Learned from Google Projects</span></i></a></span><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small;">”.</span><br /><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small;">[24]</span><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small;"> </span><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small;">Michael Kopp. “</span><a href="http://apmblog.compuware.com/2013/07/17/top-performance-problems-discussed-at-the-hadoop-and-cassandra-summits/" style="font-family: Verdana, sans-serif; font-size: small;"><i><span lang="EN-US">Top Performance Problems discussed at the Hadoop and Cassandra Summits</span></i></a><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small;">”, July 17, 2013.</span><br /><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small;">[25] Eddy Satterly. « </span><a href="https://www.youtube.com/watch?v=-N9i-YXoQBE" style="font-family: Verdana, sans-serif; font-size: small;"><i><span lang="EN-US">Big Data Architecture Patterns</span></i><span lang="EN-US"> </span></a><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small;">».</span><br /><span lang="EN-US" style="font-family: Verdana, sans-serif; font-size: x-small;">[26] Paul Ohm. “<i>Broken Promises of Privacy: Responding to the Surprising Failure of Anonymization</i>”. </span><i style="font-family: Verdana, sans-serif; font-size: small;"><a href="http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1450006##" style="font-family: Verdana, sans-serif; font-size: small;">UCLA Law Review, Vol. 57, p. 1701, 2010</a></i><br /><span style="background-color: white; font-family: Verdana, sans-serif; font-size: x-small;">[27] CIGREF, « </span><a href="http://images.cigref.fr/Publication/2013-Big-Data-Vision-grandes-entreprises-Opportunites-et-enjeux-CIGREF.pdf" style="background-color: white; font-family: Verdana, sans-serif; font-size: small;"><i>Big Data : La vision des grandes entreprises</i> </a><span style="background-color: white; font-family: Verdana, sans-serif; font-size: x-small;">», 2013</span></div><div class="MsoNormal"><br /></div><div class="MsoNormal" style="text-indent: 0px;"><b style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US"><span style="font-size: large;">2. Big Data as a new way to extract value from data</span></span></b></div><div class="MsoNormal" style="text-indent: 0px;"><b style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US"><br /></span></b></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">As introduced earlier, the key idea is to forget about causation, or trying to extract knowledge from data mining. A large part of Viktor Mayer-Schönberger and Kenneth Cukier’s book (chapter 5) is dedicated to the difference between causation and correlation. But what we found when we heard about “real big data systems”, such as those from Google or Criteo, is that these systems are anything but static data mining systems that aimed at producing knowledge. They are dynamic systems that constantly evolve and “learn” from the data, in a controlled loop. Most big data statistical tools, such as logistic regressions, or machine learning algorithms are still looking for correlations, but these correlations are not meant to hold intrinsic value, they are input for action, whose effect is measured in real-time. Hence knowing the “why” of the correlation, knowing if there is a causation, a reverse causation, a complex dependency circle which is the signature of complex systems … does not really matter. Nor does it matter (as much as in the past) to be assured that the correlation is stable and will last in time. The correlation detection is embedded into a control loop that is evaluated though the overall process financial result.<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;"><br /></span></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">This Big Data approach towards data mining is about practice and experiments. It is also broader in scope than statistics or data mining, since the performance comes from the whole “dynamic system” implementation. It is also easy to deduce from this, which we do in our report with more details, that building such a Big Data system is a team effort, with a strong emphasis on technology and distributed systems and algorithms.  From a marketing perspective, the goal is no longer to produce “customer knowledge” – which meant to understand what the customer wants – but to build an adaptive process which leads to better customer satisfaction – which is actually less ambitious. If we consider the Walmart example that is detailed in the previously mentioned book [5], analyzing checkout receipts produces correlations that need not to be thought as “customer insights”. There is no need to find out why there is a correlation between purchases of diapers and beer packs. It is enough to put these two items close by and see if sales improve (which they did). In the virtual world of the Web, testing these hypotheses becomes a child play (no physical displacement necessary).<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;"><br /></span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">There is a natural risk of error if one takes these correlations out of their dynamic-loop context and tries to see them as prediction. Our different experts, including the speakers invited for the 2013 </span><a href="http://www.naefrontiers.org/Symposia/EU-USFOE/22791/32009.aspx"><i><span lang="EN-US">Frontiers of Engineering</span></i></a><i><span lang="EN-US"> </span></i><span lang="EN-US">conference – in particular Jeff Hammerbacher from Cloudera et Thomas Hofmann from Google – were adamant about the fact that “Big Data produces information that one does not really understand”, which creates the risk of poor utilization. This is similar in a sense to the phenomenon of « </span><a href="http://en.wikipedia.org/wiki/Spurious_correlation"><i><span lang="EN-US">spurious correlations</span></i><span lang="EN-US"> </span></a><span lang="EN-US">», which says that one analyses a large cloud of data points with a very large number of variables, one finds statistically a large number of correlations without any meaningful significance. A great example of avoiding such pitfall is given by the “Google Flu Trends” (GFT) story. When they analyzed search requests that used works linked to Flu, Google researchers found that they could forecast flu epidemics propagation with a good level of accuracy. This claim was instantly absorbed, amplified and orchestrated as a proof of Big Data greatness. Then more detailed analysis [15] [16] showed the limits and the shortcoming of GFT. The article that is publish on Harvard’s blog [15] is actually quite balanced. Although it is now clear that GFT is not a panacea and shows more errors than other simpler and more robust forecasting methods, the articles also states that : « <i>The initial vision regarding GFT – that producing a more accurate picture of the current prevalence of contagious diseases might allow for life-saving interventions – is fundamentally correct, and all analyses suggest that there is indeed valuable signals to be extracted</i> ». <o:p></o:p></span></span></div><div class="MsoNormal"><br /></div><div class="MsoNormal" style="text-indent: 0px;"><span style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US">3<span style="font-size: large;">. <b>Data is the new code</b></span></span></span></div><div class="MsoNormal" style="text-indent: 0px;"><span style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US"><b><br /></b></span></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">This great catch phrase was delivered to us by <a href="http://www.henriverdier.com/">Henri Verdier</a>, one of the many experts that was interviewed by the ICT commission. When Google’s teams look at a new startup, they compute the valuation mostly from the volume and the quantity of data that has been collected, with much less regard for the code that has been developed. Data valuation comes both from the difficulty to collect the data and its estimated future usage potential. Code is seen as an “artefact” that is linked to data, which is both destined to change and easy to replace. In this new world of Big Data, code is conceptually less important than the data it applied to. It is less important because it changes constantly, because it is made of simple sub-linear algorithms (the only ones that can be run onto petabytes of data) and because it is the result of a learning loop (simple algorithms in their principles, but zillions of parameters that require to be fine-tuned through experiments). To caricature the reasoning, Google could tell to these young startups : “I will buy your data and I will re-grow the code base using our own methods”.<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;"><br /></span></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">This new way of programming does not apply only to new problems and new opportunities! This approach may be used to re-engineer more classical “information systems” through the combined application of commodity computing, massively parallel programming and open source data distribution software tools. This combination helps win one or two orders of magnitude with respect to cost, as was shown to us with numerous examples. In the previously mentioned book [5], one may learn about the VISA example, where Big Data technology was used to re-build an IT process with spectacular gains in cost and throughput. This “new way of programming”, centered on data, may be characterized in three ways:<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"></div><ul><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">Massively parallel programming because of the distribution of very large amount of data. The data distribution architecture becomes the software architecture because, as the volume grows, it becomes important to avoid “moving data”.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">Sub-linear algorithms (whose compute time grows slower than the amount of data that they process) play a key role. We heard many great examples about the importance of such algorithms, such as the use of Hyperloglog counters in the computation of <a href="http://arxiv.org/pdf/1111.4570.pdf">Facebook social graph diameter</a>.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">Algorithms need to be adaptive and tuned incrementally from their data. Hence machine learning becomes a key skill when one works on a very large amount of data.</span></li></ul><br /><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">During the 2013 FoE conference, Thomas Hoffman told us that “</span><i><span lang="EN-US">Big data is getting at the core of computer science”. </span></i><span lang="EN-US">This means that all current problems that receives the attention of today computer scientists, such as artificial intelligence, robotics, natural language processing, behavioral learning, and so on, all require the combination of these three characteristics: need for massive hence distributed computing power, huge amounts of data and machine learning to grow better algorithms.<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US"><br /></span></span></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">This does not mean that « <i>data as the new code</i> » is a universal approach towards information system design. Massive distribution of data has its own constraints and faces fundamental (theory-proven) data architecture difficulties. They are known, for instance, as the </span><a href="http://en.wikipedia.org/wiki/CAP_theorem"><span lang="EN-US">CAP Theorem</span></a><span lang="EN-US"> or </span><a href="http://www.ics.uci.edu/~cs237/reading/files/An%20introduction%20to%20snapshot%20algorithms%20in%20distributed%20computing.pdf"><span lang="EN-US">the problem of snapshots algorithms in distributed computing</span></a><span lang="EN-US">. Simply put, it is not possible to get at the same time data consistency, high availability and fault tolerance if part of the network becomes unavailable. Big Data solutions usually pick a weakened form of consistency or availability. The logical consequence is that there remains domains – mostly related to transactions and </span><a href="http://en.wikipedia.org/wiki/ACID"><span lang="EN-US">ACID</span></a><span lang="EN-US"> requirements as well as very low latency requirements – where “more classical “ architecture are still better suited.<o:p></o:p></span></span></div><div class="MsoNormal"><br /></div><div class="MsoNormal" style="text-indent: 0px;"><span style="font-family: Verdana, sans-serif; text-indent: -18pt;"><span lang="EN-US"><span style="font-size: large;">4. <b>Conclusion</b></span></span></span></div><div class="MsoListParagraphCxSpLast"><br /></div><div class="MsoNormal" style="text-align: justify;"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">These two paradigm shifts are accompanied by changes in culture, methods and tools. To finish this post and to summarize, I would quote: <b>agile methods</b>, <b>open-source software culture</b> and <a href="http://informationsystemsbiology.blogspot.fr/2011/11/lean-it-devops-and-cloud-programming.html">DevOps</a>(continuous build, integration and delivery). It is stunningly obvious that one cannot succeed in developing the kind of closed-loop data mining systems described in Section 2, nor the machine-learning-data-driven algorithms described in section 3 without the help of agile methods.  </span><a href="http://informationsystemsbiology.blogspot.fr/2012/12/lean-scrum-agile-and-extreme-programming.html"><span lang="EN-US">Agile methods</span></a><span lang="EN-US"> advocate incremental and short batch development cycles, organized around multi-skills teams, where everyone works in a synchronous manner on the same objective.  The same argument applies to the proper use of open-source software, though it is less obvious and comes from experience. It is not about using great software available for free, it is more about using continuously evolving software that represents the bleeding edge of big data technology. It is even more about the open source culture that fits like a glove to the concept of <a href="http://informationsystemsbiology.blogspot.fr/2012/07/lean-software-factory.html">software factories</a> (the topic of another post !). To succeed in this new software world, you need to love code and respect developers (and yes, <i>I am aware of the paradox that this may cause together with the “data is the new code” motto</i>). Last, there is no other way to produce continuously evolving code (which is implied by both these paradigm shifts, but is also true in the <a href="http://organisationarchitecture.blogspot.fr/2014/03/entreprises-numeriques-et-homeostasie.html">digital world</a>) than switching to continuous build, integration and delivery, as exemplified by <a href="http://en.wikipedia.org/wiki/DevOps">DevOps</a>. I am quoting <a href="http://devops.com/">DevOps</a> but I could also make a reference to the software factory idea (the two are closely related).<o:p></o:p></span></span></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh58ijkU2vxlswlF9CC7xEAgbKBJ8WzybxafnGEvcFP0VqeaIXDmI1x1siJW1Ff8qSLcBqvs8R0BQ2pITXExOcuRB8y46iBjjtsc2xy6q0NK-ZtZzokCFWv8XBfC1Z5UKLvzMoc90GB3Kc/s1600/LesG%C3%A9antsDuWeb.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: justify;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh58ijkU2vxlswlF9CC7xEAgbKBJ8WzybxafnGEvcFP0VqeaIXDmI1x1siJW1Ff8qSLcBqvs8R0BQ2pITXExOcuRB8y46iBjjtsc2xy6q0NK-ZtZzokCFWv8XBfC1Z5UKLvzMoc90GB3Kc/s1600/LesG%C3%A9antsDuWeb.jpg" /></a></div><div class="MsoNormal" style="text-align: justify;"><br /></div><div class="MsoNormal" style="text-align: justify;"><br /></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">Not surprisingly, the reader who is familiar with “Les Géants du Web” [19] from Octo will recognize the culture which is common to the “Web Giants”, that is, the companies who are the most successful in the digital world, such as Amazon, Google or Facebook. There is no surprise because these companies are also amongst the world leaders in leveraging the promises of Big Data. Agile (hence collaborative) development is critical to Big Data which requires to mix computer science, information technology, statistical and domain matter (business) skills. Because Big Data requires to work on the “real” (large) sets of data, it means a strong collaboration between IT operations and development. This is made even more critical by the paradigm shift described in Section 2, since algorithmic development and tuning is embedded into an operation cycle, which is an obvious call for DevOps.<o:p></o:p></span></span></div><div class="MsoNormal" style="text-align: justify;"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;"><br /></span></span></div><div class="MsoNormal"><span lang="EN-US"><span style="font-family: Verdana, sans-serif;">I will conclude with a few of the recommendation from the NATF report:<o:p></o:p></span></span></div><div class="MsoNormal"></div><ul style="text-align: left;"><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">Big Data is much more than new opportunities to do new things. Fueled by a technology shift that is caused by drastic price drops (storage and computing), Big Data paradigm causes a disruption about how to build information systems.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">Massive parallelism and huge volumes of data are bringing a new way of programming that is urgent to learn, and to teach. This goes for companies as well as universities or engineering schools.</span></li><li><span lang="EN-US" style="font-family: Verdana, sans-serif; text-indent: -18pt;">The old world of cautious “analyze/model/design/run” waterfall linear projects is in competition with a new world of systemic loops “experiment/learn/try/check”.  This is true for science [21] as well as for business. Hence, Big data’s new paradigms needs to be taught in business schools as well as in engineering schools.</span></li></ul><br /><div class="MsoNormal"><span style="font-family: Verdana, sans-serif;"><span lang="EN-US">Readers who are familiar with  </span><a href="http://www.la-fabrique.fr/Actualite/le-veritable-enjeu-des-big-data/"><span lang="EN-US">Francois Bourdoncle</span></a><span lang="EN-US">’s theses on Big Data will recognize them in these recommendations, which is quite natural since he was one of the experts audited by the ICT commission from the NATF.<o:p></o:p></span></span></div><div class="MsoNormal"><br /></div><br /><div class="MsoNormal"><br /></div></div>

Big Data hides more than one paradigm shift

This post originates from a report which I wrote this summer for the NATF , which proposes a summary of what the ICT commission learned from...

Read more »