Many companies have invested millions in building a successful BI / EDW and are investing in advanced analytics for the future. But the mystery remains about the data quality. Though glaring DQ issues might be contained through constant backend data corrections or through exception handling, many organizations still faces the challenge of poor data quality.
Source: Information Week
The reason Data Quality does not get addressed in many organizations because of several reason. Typically you find:
- The IT organization manually corrects the data issues over and over
- Business takes the report and adds/ modifies the data for further use
- Reports are just to verify basic information, real data resides in someone’s spreadsheet
The Future of Big Data
With some guidance, you can craft a data platform that is right for your organization’s needs and gets the most return from your data capital.
So the problem gets buried in various facets of the organization. Everybody knows the problem but no one will step up to own it or sponsor to fix it permanently. The more efficient IT is, harder it is to build a business case for DQ tools or initiating DQ projects.
Having a Data Governance organization becomes very critical in bringing the business and IT together. This is the forum where business and IT can work together to solve the DQ issues and define the ownership and accountability. A day a month of cleaning up each person who uses the data adds up quickly in terms of hours and not to mention the data discrepancies due to manual changes to data.
Matured organizations understand the DQ issues and implements the DQ as part of the overall development / operations. It is an expensive affair if the DQ goes unchecked. One time cleanup of data will slowly decay over time to right where we started. Investing in setting up DQ metrics, data ownership and other quality related policies enabled by appropriate tools is the right way to solve the Data Quality issues. DQ does not mean perfect data but good enough data to do the analysis for right decision-making.
