Data & Intelligence

Big Data – Organizational Challenges

My previous blog post was about Enterprise Search.  Within my research, I continuously came across the term Big Data.  Obviously, I had heard of this before, but I was not well versed on what it actually meant.  While I was waiting at the airport, I saw a Harvard Business Review (HBR) magazine staring at me, with the title “Getting Control of BIG DATA.”  And with that I knew it was time for me to better understand what Big Data was all about and I thought I would use this blog to write about it.  My colleague, Dominic Sagar wrote a great blog about Big Data in a previous post – What is Big Data?.  Therefore, I thought I would use this post to expand on what he wrote and summarize the HBR article – Big Data: The Management Revolution.

“You can’t manage what you don’t measure” – W. Edward Deming was credited with this saying.  With the recent explosion of digital data, management can begin to measure and manage an immense amount of data.  In my experience, the mere mention of Big Data gets many executives excited about the conversation and they want “it”, regardless of what “it” is.  So, what is “it” and how does “it” affect an organization from a cultural and managerial perspective?

Key Differences of Big Data

So, what makes Big Data unique in comparison to what we are already doing within our Analytics/BI Organizations today?

  1. Volume: The article states that, “as of 2012, about 2.5 exabytes of data are created each day, and that number is doubling every 40 months or so.  For instance, Walmart collects more than 2.5 petabytes of data every hour from its customer transactions.”  This allows more and richer data sets to be analyzed that were ever imaginable.  Understanding how to use this data is where art meets technology – enter in the Data Scientist – Data Scientist: The Sexiest Job of the 21st Century.
  2. Velocity: Near real-time analytics allows for real-time decision making to provide a competitive advantage over competitors.  Additionally, decision making can incrementally improve as more and more data is consumed to become a Learning Organization.
  3. Variety: Big Data comes in all different shapes and sizes – blog posts, images, SMS, geopositioning information, tweets, etc.  Traditional databases cannot support these forms of data.  However, technologies such as Hadoop have made the storing of Big Data economical.  As Google’s director of research, Peter Norvig, puts it: “We don’t have better algorithms. We just have more data.”

Data-Driven Organizational Performance

Data Intelligence - The Future of Big Data
The Future of Big Data

With some guidance, you can craft a data platform that is right for your organization’s needs and gets the most return from your data capital.

Get the Guide

Where’s the evidence that using big data intelligently will improve business performance?  The authors realized that businesses are aware of the importance of capturing Big Data, but there was no concrete evidence as to why they should.  So, they conducted their own investigation in which they completed structured interviews with executives at 330 North American companies asking about their organizational and technology management practices, and gathered performance data from their annual reports and independent sources.  According to the study, “Not everyone was embracing data-driven decision making.  One relationship stood out: the more companies characterized themselves as data-driven, the better they performed on objective measures of financial and operational results. In particular, companies in the top third of their industry in the use of data-driven decision making were, on average, 5% more productive and 6% more profitable than their competitors.”

Cultural Shift – Data-Driven, Decision Making Culture

As with any new change, the shift to a data-driven, decision making culture is a huge undertaking.  The article mentions two major challenges.

  1. Muting the HiPPOs (High-Paid Person’s Opinion): this is probably the most difficult challenge in that many organizational decisions are led by people in this role.  It seems that people assume that the higher someone is paid, means that their decisions are golden and organizations defer to them for the final vote.  While, these people definitely have earned their opinions to be heard (I won’t get into the compensation debate), they should not necessarily lead every decision.  Decisions now need to be led by the data.  Big Data allows the measurement of many different metrics that can replace the intuition based decision making.
  2. New Roles need to be established: to assist in the transition from intuition to data-driven decision making, the article identifies two simple techniques:
  • “First, they can get in the habit of asking “what do the data say?” when faced with an important decision and following up with more-specific questions such as “Where did the data come from?”, “What kinds of analyses were conducted?,” and “How confident are we in the results?”
  • “Second, they can allow themselves to be overruled by the data; few things are more powerful for changing a decision-making culture than seeing a senior executive conceded when data have disproved a hunch.”

I feel that the above roles are noble, I think seeing those in action will take many years of transition.  Perhaps, the “old-way” needs to retire before “fresh thinking” can rule the C-suites.  Domain experts still remain experts and their knowledge will be vital as their subject areas become the focus of Big Data.

Managerial Challenges

Perhaps the most difficult challenge in the shift to a data-driven, decision making culture are the managerial challenges.  Improperly managing these challenges will stop any Big Data project in its tracks no matter how technologically advanced the project may be.  The article lists five management challenges:

  1. Leadership: much like any good leadership team in any situation, they need to set clear goals, define what success looks like, and ask the right questions.  The important point here is that everything must change and the clear communications of the changes is imperative in order to shift the culture.
  2. Talent Management: a new data-scientist position is being generated due to this shift.  This is a new area for many organizations and the resource pool is limited.  This position must be able to speak the language of Big Data statistics and business in order to help leaders reformulate their challenges in ways that Big Data can tackle.
  3. Technology: the technologies are new and in some cases exotic.  However, the available tools are increasing and therefore, reducing the costs especially because many of the available tools are open-source.  Hadoop, the most commonly used framework, combines commodity hardware with open-source software.
  4. Decision Making: Big Data is shifting where the organizational expertise used to be and the challenge here is to bring together people who understand the problem with the people who know where the data resides and have the problem-solving abilities.
  5. Company Culture: “The first question a data-driven organization asks itself is not “What do we think?” but “What do we know?”  Organizations need to honestly asses their capabilities and where they stand as a data-driven organization.  Organizations need to move from the intuition based decision-making, to a data-driven organization.

Final Thoughts

Organizations often consider themselves to be a data-driven organization and I am sure that many actually are.  However, with the recent growth in data availability, Big Data needs become a priority for organizations and the new data which is available needs to be leveraged.  Now is the time to begin thinking about Big Data and how it can be used as a competitive advantage for your organization.  You do not need to have the expertise…yet.  But you need to have the desire to mature and grow as an organization.  The business and technology payoff are unmistakable.  “The evidence is clear: data-driven decisions tend to be better decisions. Leaders will either embrace this or be replaced by someone who does.”


About the Author

More from this Author

Thoughts on “Big Data – Organizational Challenges”

  1. Thanks for the informative article, Brett. Dom, your article was informative as well.

    Yes, I’ve been hearing a lot about “Big Data” from various sources and your blog has sparked an interest. Particularly because Big Data is getting away from my specialty of Data Warehousing and ETL.

    Where do you guys see this going? What tools are out there? Does this mean that ETL and Star Schema’s etc are soon to fade with so much data available? It’s very difficult to find the relationships with so much “variety” or data.

  2. Frank Nepomuceno

    Great post Brett,
    A couple of notes:
    I think ETL will change for some use cases to Extract, MapReduce, Load.
    Managerial Challenges, for large legacy (read old) organizations, any change is difficult. “Intuitive judgement vs. It’s the data stupid!”
    What to do with big data, what does it mean, is it actionable? Extract meaning from data, the role of the data scientist becomes more evident.
    Speed of delivering data. Need the data now, know the now, is the data actionable or is it too late.

  3. Hi Dee, thank you for your comment. I see Big Data becoming an integral part of large organizations. It will become one more source of information that we need to incorporate into our reporting solutions. As for technologies, all of the big players have a Big Data solution – IBM, Oracle, Google, etc. I have also come across some great predictive analytics examples using SPSS (IBM) or SAS. The “relationship” and “variety” is where the Data Scientists come into play. They will be the new Rockstars of IT.

  4. Brett Baloun Post author

    Hi Frank, thank you for your comment and insight. In the article that I refer to in my blog, there was a great example about how an airline was able to better predict arrival times by ingesting “real-time” gps and weather data. The end result was millions in savings. Additionally, it mentioned how Sears was able to reduce marketing campaigns from weeks to days using Big Data.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe to the Weekly Blog Digest:

Sign Up