It seems as though every large organization these days is either conducting a Big Data Proof of Concept (POC) or considering doing one. Now, there are serious questions as to whether this is even the correct path towards adoption of Big Data technologies, but of course for some potential adopters it may very well be the best way to determine the real value associated with a Big Data solution.
This week, Bill Busch provided an excellent webinar on how organizations might go through the process of making that decision or business case. For this exploration, we will assume for the sake of argument that we’ve gotten past the ‘should we do it’ stage and are now contemplating what to do and how to do it.
Capability Evolution tends to follow a familiar path…
The Future of Big Data
With some guidance, you can craft a data platform that is right for your organization’s needs and gets the most return from your data capital.
Big Data POC Assumptions:
Everything starts with assumptions – and there are a number of good ones that could be considered universal for Big Data POCs (applicable in most places), these include the following:
- When we say ‘Big Data’ what we really mean is multiple potential technologies and maybe even an entire technology stack. The days of Big Data just being entirely focused on Hadoop are long gone. The same premise still underlies the growing set of technologies but the diversity and complexity of options have increased almost exponentially.
- Big Data is now much more focused on Analytics. This is a key and very practical consideration – re-hosting your data is one thing – re-envisioning it is a much more pragmatic or perhaps more tangible goal.
- A Big Data POC is not just about the data or programming some application or even just the Analytics – it’s about a “Solution.” As such it ought to be viewed and managed the way your typical IT portfolio is managed – and it should be architected.
- The point of any POC should not be to prove that the technology works – the fact is that a lot of other people have already done that. The point is determining precisely how that new technology will help your enterprise. This means that the POC ought to be more specific and more tailored to what the eventual solution may look like. The value of having the POC is to identify any initial misconceptions so that when the transition to the operational solution occurs it will have a higher likelihood of success. This is of course the definition of an Agile approach and avoids having to re-define from scratch after ‘proof’ that the technology works has been obtained. If done properly, the POC architecture will largely mirror what the eventual solution architecture will evolve into.
- Last but not least, keep in mind that the Big Data solution will not (in 95% of the case now anyway) replace your existing data solution ecosystem. The POC needs to take that into account up front – doing so will likely improve the value of the solution and radically reduce the possibility of running into unforeseen integration issues downstream.
Perhaps the most important consideration before launching into your Big Data POC is determining the success criteria up front. What does this mean? Essentially, it requires you to determine the key problems that the solution is targeted to solve and coming up with metrics that can be objectively obtained from the solution. Those metrics can be focused both on technical and business considerations:
- A Technical metric might be the ability update a very large data set based on rules within a specified timeframe (consistently).
- A Business metric might be the number of user-defined reports or dashboard visualizations supported.
- And of course both of these aspects (technical and business capability) would be governed as part of the solution.
Without the POC success criteria it would be very difficult to determine just what value adopting Big Data technology might add to your organization. This represents the ‘proof’ that either backs up or repudiates the initial business case ROI expectation.
In my next post, we will examine the process of choosing “What to select” for a Big Data POC…
copyright 2014, Perficient Inc.