Big Data: Do you really need it? Findings and thoughts

Yeah, yeah, another post about Big Data. And, yes I know, for someone who complains about the number of posts on Big Data I’m only adding to the noise. But as I heard more and more about big data  I decided to build myself a little program that I could point at companies and track what people were saying about them. After a few months of data gathering I’ve found for a majority of companies most of the data I gathered was just noise. In fact in a large majority of cases (>85%) over half of what the social data out there was put out about a company was put out by the companies themselves or their employees, in some cases that percentage is as high as 98%.

Revolutionize Your Business With Generative AI

From product design and software development to virtual agents, content creation, and reporting, GenAI is transforming business. Our AI experts help you unlock GenAI’s full potential and drive growth.

Let’s Get Started

If you filter that out you’re left a tiny bit of data and of that tiny bit only about half is of any use.  So now you’re essentially looking for needles in a haystack. It’s enough to make a man go home and be a family man. However, their rarity doesn’t necessarily make them valuable. Sometimes a needle is just a needle.  Which leads to an obvious question which Michael Wu made a really great post on:, “If 99.99% of Big Data is irrelevant, Why Do we need it?”

The answer? Maybe you do, maybe you don’t. Like I said in my little non-scientific study it was certainly true that most companies put out most of the noise about themselves. However, for certain companies (companies with large customer facing arms such as Hotels,  food, entertainment, etc) it certainly makes sense to make use the swell of information out there.  And certainly there’s big data that lies outside of the social sphere. Perhaps your company gathers data from the product base. That makes a lot of sense. However, for a lot of companies Big data might just not make sense yet.  In the post mentioned above Michael Wu postulates 3 scenarios to help ascertain if you’re ready for big data they are:

1. If you have access to the talent and can do it cheaply. That includes the talents to extract and analyze the relevant data in order to derive insights and value from it
2. If you are a DaaS provider and need the data to serve your customers
3. If you have specific questions, then all you really need is just the “right” data, which is usually not big at all!

I could not agree more and my initial findings back this right up.