Selecting and deploying a test management system is probably one of the hardest topics in BI mainly due to the lack of lightweight, easy to use test management frameworks well suited to BI. Most of the mature test management suites target application and, most recently, web application development. And that’s not all that useful for large databases.
However, not all is lost. There are a few possible strategies for dealing with this gap:
- Buy one of the big dogs such as the IBM Rational tools or HP’s Quality Center (formerly Mercury). Costs a bunch, major learning curve, and very powerful.
- Roll your own. Not as crazy as it sounds in this space since you already have some relevant tools in your tool-belt: your DBMS and its stored procedure language, your build system, your issues management system, and your reporting tool. By limiting your custom build to simply “gluing up” these existing tools, rolling your own test platform is a practical possibility.
- Leverage an existing unit test framework that targets your specific platform. Again, most of these aren’t very mature, but there are a handful of xUnit compliant frameworks available for every major DBMS.
Note that the above discussion is entirely focused on targeting the DBMS in tests. There are a couple reasons for this. First, in BI, the buck pretty much stops exclusively with the data. So, the vast majority of tests are going to look at data directly. What this doesn’t cover are things like end user security, report output, etc. However, testing these aspects of the system will require direct support from the specific toolset being tested, and most of the major vendors provide some path for automating unit tests of these system aspects. Some of them may require the addition of a web application testing tool to mimic user interaction with the system and luckily there are many options in this class.
Beyond the normal critical criteria for the selection of a test management system (such as compatibility with your DBMS), I encourage clients to consider:
- Integration with the continuous integration (CI) system. The framework must both be called from and provide feedback to the CI system.
- Integration with the issues management system, either directly or via the CI system.
- Modular design, especially the ability to “plug in” diverse testing subsystems to target different DBMSes, ETL platforms, BI platforms, etc. Note that I’ve not had good luck finding this type of design other than just sticking to the xUnit family of test frameworks.
All in all, test management continues to be one of the most difficult tool classes for BI. There’s just not a short and sweet answer. But that doesn’t mean you should just give up on BI test automation. It simply means this is going to be an area you’ll have to approach with a little more can-do than most!
