One of the most frequently asked questions by our customers is the roadmap to deploying a Big Data Platform and becoming a truly data-driven enterprise. Although the roadmap is different from customer to customer, at a high level, I would break it down into three phases:
Phase – 1 Building the Foundation
Just as you can’t build a house without a foundation, you can’t start down a big data path without first establishing groundwork for success. There are several key steps to prepare the organization to realize the benefits of a big data solution with both structured and unstructured data. The first step is to start with the structured data. To build the foundation, you need do the following:
- Acquire and organize structured data into a high-performance and scalable data warehouse platform
- Consolidate structured data to provide an integrated and consistent enterprise-wide view of the business
- Shorten the time that is necessary to analyze structured data based information, and decrease the number of participants who are involved in the processing of this information
- Build a common enterprise information model (semantic model), which can also be accessed via an open API, making it available to any delivery channel, thus providing a single source of truth
- Define KPIs and metrics by subject area (data mart) to allow users the ability to create complex ad-hoc queries
- Set up a scalable security infrastructure to automate authentication (access) and authorization (visibility) process
- Provide training and workshops to facilitate user adoption
- Build a business intelligence support team
- Build a business intelligence competency center
Phase – 2 Setting Up the Big Data Infrastructure
Once the foundation is complete, you need to frame the house before running the electrical and plumbing. For your big data journey, Phase 2 means shifting the focus to the big data infrastructure. This includes the following:
- Equip the business intelligence layer with advanced analytics, in-database statistical analysis, predictive analysis, and advanced visualization, on top of the traditional components such as reports, dashboards, and queries
- Deploy information discovery tools to understand “why” and provide users the ability to investigate and understand cause by exploring relationships between structured, semi-structured, and unstructured data sources
- Set up a scorecard and strategy management to provide users the capabilities to communicate strategic goals and monitor progress over time
Phase – 3 Predicting the Future
The finishing touches are what make a house your home – the paint, the carpet, the fixtures. To get the most out of your big data investment you need to leverage the core information architecture principles and practices implemented in previous phases and further enhance it to:
- Bring different data sources with large data sets together for timely and valuable analysis that are challenging to store, search, share, visualize, and analyze
- Provide data integration capability that will cover the entire spectrum of velocity and frequency and be able to handle the extreme and ever-growing volume requirements
- Set up an architecture so the business users do not see a divide. They don’t even need to be made aware that there is a difference between traditional transactional data and big data. Make the data and analysis flow seamless for users to navigate through various data and information sets, test hypothesis, pattern analysis, and be able to make informed decisions.
- Provide the ability to correlate with other structured and unstructured enterprise data using powerful statistical tools allowing uses to find the needle in the haystack, and ultimately helping to predict the future
- Provide a Complex Event Processing engine to analyze stream data in real time
Till now, this is the most comprehsensive article describing a Big Data platform roadmap. Looking forward for more insights from your side on Big Data and its infrastructure.