Big Data – Organizational Challenges

My previous blog post was about Enterprise Search.  Within my research, I continuously came across the term Big Data.  Obviously, I had heard of this before, but I was not well versed on what it actually meant.  While I was waiting at the airport, I saw a Harvard Business Review (HBR) magazine staring at me, with the title “Getting Control of BIG DATA.”  And with that I knew it was time for me to better understand what Big Data was all about and I thought I would use this blog to write about it.  My colleague, Dominic Sagar wrote a great blog about Big Data in a previous post – What is Big Data?.  Therefore, I thought I would use this post to expand on what he wrote and summarize the HBR article – Big Data: The Management Revolution.

“You can’t manage what you don’t measure” – W. Edward Deming was credited with this saying.  With the recent explosion of digital data, management can begin to measure and manage an immense amount of data.  In my experience, the mere mention of Big Data gets many executives excited about the conversation and they want “it”, regardless of what “it” is.  So, what is “it” and how does “it” affect an organization from a cultural and managerial perspective?

Key Differences of Big Data

So, what makes Big Data unique in comparison to what we are already doing within our Analytics/BI Organizations today?

1. Volume: The article states that, “as of 2012, about 2.5 exabytes of data are created each day, and that number is doubling every 40 months or so.  For instance, Walmart collects more than 2.5 petabytes of data every hour from its customer transactions.”  This allows more and richer data sets to be analyzed that were ever imaginable.  Understanding how to use this data is where art meets technology – enter in the Data Scientist – Data Scientist: The Sexiest Job of the 21st Century.
2. Velocity: Near real-time analytics allows for real-time decision making to provide a competitive advantage over competitors.  Additionally, decision making can incrementally improve as more and more data is consumed to become a Learning Organization.
3. Variety: Big Data comes in all different shapes and sizes – blog posts, images, SMS, geopositioning information, tweets, etc.  Traditional databases cannot support these forms of data.  However, technologies such as Hadoop have made the storing of Big Data economical.  As Google’s director of research, Peter Norvig, puts it: “We don’t have better algorithms. We just have more data.”

Data-Driven Organizational Performance

Revolutionize Your Business With Generative AI

From product design and software development to virtual agents, content creation, and reporting, GenAI is transforming business. Our AI experts help you unlock GenAI’s full potential and drive growth.

Let’s Get Started

Where’s the evidence that using big data intelligently will improve business performance?  The authors realized that businesses are aware of the importance of capturing Big Data, but there was no concrete evidence as to why they should.  So, they conducted their own investigation in which they completed structured interviews with executives at 330 North American companies asking about their organizational and technology management practices, and gathered performance data from their annual reports and independent sources.  According to the study, “Not everyone was embracing data-driven decision making.  One relationship stood out: the more companies characterized themselves as data-driven, the better they performed on objective measures of financial and operational results. In particular, companies in the top third of their industry in the use of data-driven decision making were, on average, 5% more productive and 6% more profitable than their competitors.”

Cultural Shift – Data-Driven, Decision Making Culture

As with any new change, the shift to a data-driven, decision making culture is a huge undertaking.  The article mentions two major challenges.

1. Muting the HiPPOs (High-Paid Person’s Opinion): this is probably the most difficult challenge in that many organizational decisions are led by people in this role.  It seems that people assume that the higher someone is paid, means that their decisions are golden and organizations defer to them for the final vote.  While, these people definitely have earned their opinions to be heard (I won’t get into the compensation debate), they should not necessarily lead every decision.  Decisions now need to be led by the data.  Big Data allows the measurement of many different metrics that can replace the intuition based decision making.
2. New Roles need to be established: to assist in the transition from intuition to data-driven decision making, the article identifies two simple techniques:

• “First, they can get in the habit of asking “what do the data say?” when faced with an important decision and following up with more-specific questions such as “Where did the data come from?”, “What kinds of analyses were conducted?,” and “How confident are we in the results?”
• “Second, they can allow themselves to be overruled by the data; few things are more powerful for changing a decision-making culture than seeing a senior executive conceded when data have disproved a hunch.”

I feel that the above roles are noble, I think seeing those in action will take many years of transition.  Perhaps, the “old-way” needs to retire before “fresh thinking” can rule the C-suites.  Domain experts still remain experts and their knowledge will be vital as their subject areas become the focus of Big Data.

Managerial Challenges

Perhaps the most difficult challenge in the shift to a data-driven, decision making culture are the managerial challenges.  Improperly managing these challenges will stop any Big Data project in its tracks no matter how technologically advanced the project may be.  The article lists five management challenges:

1. Leadership: much like any good leadership team in any situation, they need to set clear goals, define what success looks like, and ask the right questions.  The important point here is that everything must change and the clear communications of the changes is imperative in order to shift the culture.
2. Talent Management: a new data-scientist position is being generated due to this shift.  This is a new area for many organizations and the resource pool is limited.  This position must be able to speak the language of Big Data statistics and business in order to help leaders reformulate their challenges in ways that Big Data can tackle.
3. Technology: the technologies are new and in some cases exotic.  However, the available tools are increasing and therefore, reducing the costs especially because many of the available tools are open-source.  Hadoop, the most commonly used framework, combines commodity hardware with open-source software.
4. Decision Making: Big Data is shifting where the organizational expertise used to be and the challenge here is to bring together people who understand the problem with the people who know where the data resides and have the problem-solving abilities.
5. Company Culture: “The first question a data-driven organization asks itself is not “What do we think?” but “What do we know?”  Organizations need to honestly asses their capabilities and where they stand as a data-driven organization.  Organizations need to move from the intuition based decision-making, to a data-driven organization.

Final Thoughts

Organizations often consider themselves to be a data-driven organization and I am sure that many actually are.  However, with the recent growth in data availability, Big Data needs become a priority for organizations and the new data which is available needs to be leveraged.  Now is the time to begin thinking about Big Data and how it can be used as a competitive advantage for your organization.  You do not need to have the expertise…yet.  But you need to have the desire to mature and grow as an organization.  The business and technology payoff are unmistakable.  “The evidence is clear: data-driven decisions tend to be better decisions. Leaders will either embrace this or be replaced by someone who does.”