Way back at my first full time job, maintaining the system automation program was one of my primary duties. It was written in C, could launch one thing at a time, and documented dependencies in a cryptic text format. And I loved it because the alternates were way worse.
Fast forward to today where system automation has become a non-issue in most ways. The tools are robust, handle a laundry list of scenarios, and have all kinds of notification options including Twitter. This one’s going to be short and sweet.
Here’s what I want in an automation tool:
- Execute jobs locally and remotely across all machine instances in my environment.
- Handle dependencies triggered by internal jobs or by any number of external factors including time, file availability, service availability, web service query result, phase of the moon, etc.
- Notify me in any way I can dream up, including at least email, issue/ticket, pager/phone, SMS, and changing the stoplight icon on the BI dashboard.
- A nice visualization of the overall state of the system that can be included in some kind of operations dashboard.
- Is script-able to allow scheduling to be deployed in the same way as the remainder of the system.
Easy enough. Actually, if you’re in an organization of any size, there’s probably an enterprise class tool already out there that you can (should) piggyback on – something like CA Autosys or IBM/Tivoli Workload Scheduler or Cisco Tidal. If you’ve got one of these, great.
If not, there’s a few options:
- Let each tool schedule itself. For instance, you might use SQL Agent (SQL Server) for ETL and other data manipulation and Business Object’s internal scheduler for reports. Works, but doesn’t meet many of the above criteria.
- Buy one of the above tools or something similar.
- Go open source with something like TORQUE.
- Patch together a scheduling system using your platforms task scheduler (“task scheduler”, cron, etc.) and a build system such as make, msbuild, etc.
This is a tool class where I really hope you don’t have to spend much time or effort. If you can afford it, buy one of the commercial tools and move on to more interesting problems. Otherwise, you’ll end up sinking a lot of time and energy into this rather uninteresting (but critical nonetheless) aspect of you BI system.
