The Future of Big Data
With some guidance, you can craft a data platform that is right for your organization’s needs and gets the most return from your data capital.
I’m always striving to lower the wall between design and implementation. In BI, this means taking as much of the data architect and business analyst’s work and applying it directly to the data management process (without the developer or administrator middlemen). Here’s some opportunities:
- Model builds – Your modeling tool should be able to produce DDL that can be executed on your target database without modification. If not, fix it (new config, new tool, etc.). Then add that DDL to your CI or automated build system and stand back.
- Test Data – Create reusable test data sets and configure your build system to be able to load them as part of your automated test suite.
- Tests – Pick a testing tool that allows designers to specify functionality via tests (the basis for test driven development). Add the tests to the build.
- Data validation – Pick an accessible expression language and start specifying data validation rules in this (executable) language. ETL loads the rules from metadata at run time. The cost of validating data just dropped dramatically! And, the reliability of the system improved greatly since the only ETL code that needs testing is the single dynamic validation package.
Taking these steps means that your developers get to start putting their time, effort, and brain power towards the difficult data management problems that require a developer’s attention. No more copy, paste, tweak, repeat days for ETL developers makes everyone happy!
