Recent articles and the technology hype around Big Data, is forcing big companies as well as medium size companies to do something with this technology. Lot of companies have kicked off Big Data projects albeit in silos. New titles like Data Scientists and the cool factor has increased the expectations of value, the Big Data might bring. If we know our Data warehouse history, one can recall how Data Warehouse started big, went into hiding, and came back as Business Intelligence, later.
Data warehouse failed because of false promises, bad estimates of time and money, utopian dream and high expectations without tangible/ trustworthy output. DW has to go through many makeovers to get accepted and be part of the Enterprise solution. I see similar situation developing if companies do not pay attention to the business case, end results and bringing all the necessary players to the Big Data party.
In my opinion, any amount of insight you can get from Big Data is only good if one can link that to the Enterprise Data (read Master data, Transaction data). Having the context is the key, to leverage the information. For example, if Big Data analysis confirms that 20-30 year old male population is spending money on similar products, then:
- Does the enterprise data has current customer’s demographics?
- What is the accuracy of contact details? Is it possible to run a successful promotion for that demographics?
- How to reach these prospects and include them in the prospects/ customer Master Data?
- What if we increase the sales by 5% – would that mean production issues?
- How old is this data? Can we verify the patterns in the past and the impact?
Focusing only on Big Data, will only get 50% of the result. Knowing what is available in the EDW and improving the data quality, will yield better results when combined with the external Big Data. It is very important to build a strategy to include Big Data in the overall scheme of analytics, rather than building solutions in silos.
You have to tie that data to real actionable insights. Knowing that XYZ demographic is spending money is great, but where are they spending money? What upsells work the best? Which products are moving the quickest versus being worth the most?