Search technology has been around for more than a decade now, but they have been used to index web files and to help users find what they are looking for. The search technology when used in the enterprise BI context brings two new capabilities to BI. First, search analyses data behind the scene that traditional BI cannot, including unstructured content like documents , social media updates, RSS/twitter feeds and other highly diverse data. These are usually information that are hard to get into the warehouse and takes huge efforts to structure the data. Data discovery can be thought as a searching engine combined with unstructured and structured content in the enterprise. Data Discovery tools speed the time to insight through the use of visualizations, best practices in visual perception, and easy exploration.
This is a 3 part series providing an overview of data discovery, current data discovery tools and how data discovery can complement BI.
Overview – The Need for data discovery
BI answers the “what” of the information and gives us insights into what happened, but does not answer “why” something happened like “why did the sales decrease” or “why did the insurance claims increase”. Data discovery is the technology that helps users the answer the question beginning with “why”. Discovery as you are aware is the action or process of discovering or being discovered. Data discovery is very true to the meaning where we perform unscripted exploration or a quest on the data to find the truth.
Data Discovery are tools and technologies that are not replacements for the traditional BI modules but help in filling gaps that traditional BI cannot address. We are always aiming to go to the top of the knowledge hierarchy by using newer technologies and paradigm in making sense of the data. Data discovery is another step in achieving the ultimate objective of gaining wisdom. It brings us a little closer to the goal.
As a BI analyst my first thought while understanding the need for such a tool was with pessimism.
Why do i need a new tool when the business users can use adhoc query/reporting tools to answer their questions ?
Adhoc query can answer only a part of the problem in performing exploration on data, again only if the data is structured and do not work on ever growing unstructured data where most business are not able to gain valuable insights.
To answer the “why”, we need to follow new trails, identify new patterns and trends, which are immediately not apparent with traditional BI. BI is highly focused and needs structured data to begin with and requires data warehouse experts to maintain and streamline the process. Since business models are changing constantly and cannot entirely rely on data warehouses to present them with every answer they need, businesses need exploratory tools and technologies which address their needs faster and in real time. Data discovery tools help organisations index and tap into the the vast information sources available in the Data Warehouse, CRM, ERP , 3rd application data, web data , document and excel files.
In my next post in the series, i would discuss about specific use cases where data discovery tools might be useful, current data discovery tools and future trends.
