Right after I posted my blog on Spark and Hadoop, I came across this article. IBM 
made a big announcement that they are putting their weight behind Spark. They are committing more than 3,500 developers and programmers to help move Spark forward. This combined with significant support from the Big 3 Hadoop distributors (HortonWorks, Cloudera, and MapR) Spark seems to have a lot of momentum.
What is going on? Is Big Data is dead!? One can see Google articles proclaiming such. As discussed yesterday, some have said that Spark will replace Hadoop. Gartner a couple of years ago has proclaimed that Big Data has entered the Tough of Disillusionment. Now IBM has very publicly proclaimed massive support for Spark.
So, Big Data is dead, right? Well let’s investigate this a bit further. Just like any new transformational technology, at its onset Big Data has been driven by hype and promise. IT has jumped on the bandwagon because they realize that Hadoop, NoSQL, and other open source projects bring new gee-whiz technical capabilities. But technical capabilities alone do not create business value. Even Spark alone will not create business value. To create value, business focused solutions must be developed.
Up until recently, the focus of Big Data has been more on the technology side of things, especially around managing and storing this large influx of data we are seeing from mobile, Internet of Things, and social. The Big Data hype has been focused on how to ingest into Hadoop and leveraging an analytical tool, and magically you will garner high-value insights.
However real the value has been elusive to most companies. Depending on which analyst firm to whom you subscribe, only 20% of the companies
experimenting with Hadoop have deployed projects to production. Why is this number so low? Simply put, IT was driving most of the Hadoop investment. However, in 2014 there seems to be a huge shift. Datameer published on their blog, that the executive interest in Big Data shifted from predominately IT to predominately business executives. To put it another way, overhyped IT driven Big Data is what is dead. However, the business opportunity that the new data ecosystem presents is real.
With business executives finally getting the message that they need to leverage analytics in the new data ecosystem, organizations will need to bring business centric solutions to the market. THIS, is where the IBM Spark announcement is significant. IBM is committing 3500 people Spark! That is at a minimum a $350Million dollar annual investment. A pretty huge undertaking even for IBM. What is IBM doing? In my opinion two things. One is to get Spark ready for prime time. The challenge with Spark has been its enterprise maturity. A lot of promise, Spark needs to mature. Once Spark is mature, IBM will be free to build high-value business oriented applications that leverage analytics and machine learning using Spark. Big Data is not Dead – it is just becoming business focused.
