Automate everything! The mantra of the real developer – I’d rather spend 2 hours automating a task than 30 minutes of repetitive typing! Continuous integration finally legitimizes that innate desire to automate during the development process.
Continuous Integration (CI) tools automate the application build process and automated tests on a regular interval (or on demand as code are checked in). This allows for frequent feedback on some of the basic aspects of quality and can improve confidence in the overall platform’s state moving towards a release. In BI, continuous integration is especially helpful in detecting synchronization issues between data models, integration (ETL) code, semantic code (report models/universes) and reports.
CI tools generally have two parts, the controller and the scripting interface.
The controller manages builds and reports on successes and issues. Integration with the source control, issues management, and even the documentation management system are all useful features of the CI controller.
The Future of Big Data
With some guidance, you can craft a data platform that is right for your organization’s needs and gets the most return from your data capital.
The scripting interface provides developers a mechanism by which to act upon each of the platform’s components. Usability in scripting means the use of a well-known and supported scripting language such as Perl, JavaScript, vbscript, PowerShell, etc. Alternately, some CI tools rely on a dedicated build tool such as Make, Maven, or Ant to handle scripting giving more flexibility.
Effective CI tools:
- Can control all aspects of the build process through scripts, plug-ins, or black magic. If you have to sign in and pull a lever in the middle of a CI build, you’ve defeated the purpose.
- Report build breaks and test failures directly through the issues management/ticketing system with meaningful, useful messages.
- Are triggered by a variety of conditions, especially the commit of code into the source control system.
- Allow no-touch production builds that maintain the security envelope of the production system.
There are many open source and commercial CI tools and their features vary widely. None, as far as I know, are specific to BI, nor do they need to be. However, a nice user interface that helps out with the task of operating the ETL, BI delivery, DBMS, etc. tools is a nice way to go.
Tools I’ve run across, on the open source side:
- Hudson/Jenkins
- Apache Continuum
- CruiseControl
And in the commercial space:
- Atlassian Bamboo
- IBM Rational
- Microsoft Team Foundation Server (TFS)
Once again, I favor the Atlassian tool Bamboo, here simply because of its integration with the other Atlassian tools. So, if you don’t have JIRA or Confluence, ignore this. Bamboo is a very capable CI platform (although I think that Hudson and others are more configurable) with a beautiful user interface. Is supports remote job execution via remote agents and is competitively priced. And, getting it to create JIRA issues is a no-brainer.
That said, most CI tools are up to the task of automated builds. What you should be looking for is a tool that fits well within your overall environment and makes your team more productive through continuous feedback.
Hi Chris,
Do we have any automated tool available for Coding Guidelines enforcement for BI? (Somehting similar to Resharper or FxCop that is used for .NET).
Also is there any tool available to integrate into Build environment to enforce coding guidelines?
Thanks,
Ajeet
Ajeet –
Great question, but unfortunately no good answer on my end. We’ve got great guidelines enforcement in the model via the built-in “check” functionality in PowerDesigner. But nothing on the ETL code end of things.
Chris
Nice blog about tooling. There is much useful information. Thanks for sharing this informative blog.