The Open Source Movement that Beat all the Incumbents
Yesterday, Microsoft announced that it was expanding its support to the Spark technology stack by shipping a providing different versions of its R Server technology running on the popular MPP engine. This is the latest example of the level of support that the open source big data movement (Hadoop, Spark, etc) has received from enterprise software incumbents. A few years ago, the picture of fierce enterprise software competitors like Oracle, IBM, Microsoft, HP, EMC, SAP, etc universally lining up efforts behind an open source project that neither one of them control was completely unheard of. Somehow Hadoop changed everything.
From the late 90s to the late years of the past decade, many popular open source movements repeatedly failed to be widely adopted by big enterprise software vendors. In many cases (ex: MySQL, JBOSS, etc), the incumbents shipped alternative technologies with more sophisticated support and commercial distribution mechanisms. In other cases, one incumbent will create the primary commercial distribution of the technology and all its competitors will provide alternative solutions. To that, you should add that, until recently, open source was a bad word within enterprise software giants like Microsoft, SAP or HP.
5 Factors that Made Hadoop Different
There are a series of factors that contributed to the universal adoption for Hadoop and its successors in the big data space by the enterprise software incumbents. At the beginning, it seemed that the Hadoop movement was going to follow the same path of previous open source movements. Oracle and Microsoft continuously tried to ship technologies that directly competed with Hadoop and the adoption by other enterprise software incumbents was not very impressive at the beginning. However, over time a series of factors combined to make Hadoop a universal favorite of the enterprise software giants:
1. Too Big to Fight: At some point, the Hadoop developers and customer communities started growing to a level on which it made more sense for the incumbent to embrace it than to fight it.
2. The alternatives sucked: Let’s face it, Hadoop’s story would have been very differently if any of the enterprise software incumbent would have managed to ship a platform that came anywhere close to its capabilities. After a few failures, the incumbents were convinced that embracing Hadoop was a more viable economic model than to continue trying to compete.
3. The Hand of the Big Internet Companies: Typically, big data was a problem associated with big internet companies like Google, Yahoo, EBay, etc. The fact that most enterprises were looking at the internet giants as reference and the fact that those companies were incredibly supportive of Hadoop helped with the adoption of the big data stack within traditional IT departments.
4. Microsoft’s Help: Make no mistake, Hadoop’s trajectory could have been way more difficult if Microsoft would have decided to continue competing with it. By partnering with Hortonworks and providing Hadoop as a first class service in the Azure cloud, Microsoft put a stamp of approval on the big data movement that has helped catalyzed its adoption in the enterprise.
5. Startups Provided the Top Hadoop Commercial Distributions: One unique factor about the Hadoop movement is none of the top commercial distributions were owned by the enterprise software incumbents. Companies like Cloudera, Hortonworks or MapR still dominate the commercial Hadoop space. Without commercial domination, many of the incumbents to were forced to actively partner with these companies which tremendously helped with the adoption and distribution of Hadoop
Many other factors have contributed to the universal adoption and support of Hadoop by big enterprise software incumbents. Hopefully the lessons learned from the Hadoop movement would be replicated by other open source technologies in order to improve its adoption by enterprise software giants.
