Last time, I posted about how BI/DW accelerator and framework tools need to be used with care, and not as a substitute for solid requirements analysis. This time, I want to debunk a misconception that can be framed by the following question: Will Agile processes speed up my BI development timeline?
I see many situations where misconceptions about using Agile methods have gotten people and projects into a tight corner. And it’s not just in database/BI development, but in virtually every type of development. This is just one of them. But why is that? What is the disconnect here?
First, let’s just answer the question: No, Agile methods are probably not going to “speed up” any part of software development. Sorry. :-/
As to “why”, I think the confusion arises from the fact that, since doing Agile means favoring working product over documentation, you will usually see a “first draft” of part of a solution pretty early on. This “draft” may have some functionality initially but, in most cases, for any sizable solution it won’t be much. Some screens/UI components might be in place, but it’s most likely that the screens won’t actually do much or take you anywhere. The point is, this “first draft” is reviewed by the business, feedback gathered and then it’s iterated upon to build more real functionality. Build working software instead of models and document, and keep rebuilding and changing until it’s “done.”
And this gets to the core of what Agile methods are really about, which is about how to take in stride a continuously changing set of software requirements. While so-called “BDUF” methodologies lean on mechanisms for Change Control, Agile simply embraces and accepts that such changes are part of the deal. The Agile software development movement began as a response to inevitable requirements changes — not as a way to accelerate development. The early prototypes you get out of the first sprints in an Agile project do not signal that the project will finish faster. It just means that during development you favor working software more than ancillary process and/or documentation.
So then with respect to Data Warehousing/Business Intelligence projects, if Agile isn’t going to speed things up, what is it actually good for? Potentially a lot.
For instance, Kimball Method data warehouse development is already iterative, so it can be made to fit fairly well into an Agile context — albeit with some tweaking on matters like sprint length, sprint goals, etc. So this provides an interesting option for managing Kimball method development that may be a good fit for your environment.
Agile report development can also be quite worthwhile. Something I learned early in my consulting career was that it’s a lot easier to helpful responses from users if you show them something they DON’T want first. Starting with a simple working prototype of a report can be much more fruitful than starting with abstract specs and mockups.
In all, I think that Agile methods are best paired with data work in the following types of scenarios:
- Product Development – When you are starting from scratch, and have few or no solid requirements to start with, Agile is your friend. It will give you the opportunity to explore those requirements, stir up some ideas, and still come away with reasonably solid and reusable development progress. However, whether to stick with Agile beyond initial prototyping is another question — mainly due to the fact that some DW/BI development tasks can be hard to fit into the Agile context for an Enterprise-level solution.
- Maintenance Mode – If you have a mature DW/BI solution that is live and in production, Agile can lend itself well to managing maintenance workloads and ongoing changes. Agile maintenance can give IT departments greater flexibility in responding to sudden priority changes coming from the business.
Are those the only use cases? No, but that’s really a question for another post…. Bottom line: using Agile methods to develop your DW/BI solution does not automatically mean the project will run faster or deliver value sooner. But it can offer some serious non-speed-related benefits — depending on the context of the project.
Next time, I’ll wrap up this series with some ideas gettting value out of a DW/BI project as early as possible. Cheers!
