Skip to main content

Data & Intelligence

IBM’s Spark Investment is Evidence Big Data is Dead


Right after I posted my blog on Spark and Hadoop, I came across this article. IBM Big Data RIP tombstone-thumb-300x385-538made a big announcement that they are putting their weight behind Spark.  They are committing more than 3,500 developers and programmers to help move Spark forward. This combined with significant support from the Big 3 Hadoop distributors (HortonWorks, Cloudera, and MapR) Spark seems to have a lot of momentum.

What is going on? Is Big Data is dead!? One can see Google articles proclaiming such. As discussed yesterday, some have said that Spark will replace Hadoop.  Gartner a couple of years ago has proclaimed that Big Data has entered the Tough of Disillusionment. Now IBM has very publicly proclaimed massive support for Spark. 

Data Intelligence - The Future of Big Data
The Future of Big Data

With some guidance, you can craft a data platform that is right for your organization’s needs and gets the most return from your data capital.

Get the Guide

So, Big Data is dead, right?  Well let’s investigate this a bit further. Just like any new transformational technology, at its onset Big Data has been driven by hype and promise.  IT has jumped on the bandwagon because they realize that Hadoop, NoSQL, and other open source projects bring new gee-whiz technical capabilities. But technical capabilities alone do not create business value. Even Spark alone will not create business value.  To create value, business focused solutions must be developed.

Up until recently, the focus of Big Data has been more on the technology side of things, especially around managing and storing this large influx of data we are seeing from mobile, Internet of Things, and social.  The Big Data hype has been focused on how to ingest into Hadoop and leveraging an analytical tool, and magically you will garner high-value insights.

However real the value has been elusive to most companies.  Depending on which analyst firm to whom you subscribe, only 20% of the companiesIT Screen-Shot-2015-02-05-at-9.18.04-PM experimenting with Hadoop have deployed projects to production. Why is this number so low? Simply put, IT was driving most of the Hadoop investment.  However, in 2014 there seems to be a huge shift.   Datameer published on their blog, that the executive interest in Big Data shifted from predominately IT to predominately business executives.  To put it another way, overhyped IT driven Big Data is what is dead.   However, the business opportunity that the new data ecosystem presents is real.

With business executives finally getting the message that they need to leverage analytics in the new data ecosystem, organizations will need to bring business centric solutions to the market.  THIS, is where the IBM Spark announcement is significant.  IBM is committing 3500 people Spark!  That is at a minimum a $350Million dollar annual investment.  A pretty huge undertaking even for IBM.  What is IBM doing?   In my opinion two things. One is to get Spark ready for prime time.   The challenge with Spark has been its enterprise maturity.  A lot of promise, Spark needs to mature.  Once Spark is mature, IBM will be free to build high-value business oriented applications that leverage analytics and machine learning using Spark.  Big Data is not Dead – it is just becoming business focused.


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Bill Busch

Bill is a Director and Senior Data Strategist leading Perficient's Big Data Team. Over his 27 years of professional experience he has helped organizations transform their data management, analytics, and governance tools and practices. As a veteran in analytics, Big Data, data architecture and information governance, he advises executives and enterprise architects on the latest pragmatic information management strategies. He is keenly aware of how to advise and lead companies through developing data strategies, formulating actionable roadmaps, and delivering high-impact solutions. As one of Perficient’s prime thought leaders for Big Data, he provides the visionary direction for Perficient’s Big Data capability development and has led many of our clients largest Data and Cloud transformation programs. Bill is an active blogger and can be followed on Twitter @bigdata73.

More from this Author

Follow Us