Skip to main content


5 Common Pitfalls for Agile BI Projects

Having recently come off a string of Agile BI projects, I’ve made some observations about factors which can potentially hamstring such efforts.
Great (but Unrealistic) Expectations
5 Common Pitfalls for Agile BI ProjectsFirst, Agile is not a silver bullet. It will help deal with rapidly changing requirements, and it will get “something” together, rather than “nothing”. But Agile suits some situations and scenarios better than others:

  • New product development, where requirements are not explicit, change is expected, and a fixed timeline is in place
  • Maintenance situations, where tight sprint cycles are helpful in grouping bug fix or issue enhancement work, and in tracking/reporting progress
  • RAD situations where time-to-market is the key consideration

Notice that “Building a Data Warehouse REALLY FAST!” is not included in that list. It is hard to divide up Kimball method Data Warehousing into sprint cycles, and you can end up working against the grain of the process. Some BI development tasks/activities are not “chunkable” by sprint. Long (3-4 week) sprints as opposed to classical 2 week sprints can help with this. But at most complex, multiple “scrumlets” running on various internal sprint schedules might be necessary. Agile may not ultimately provide the time- and cost- oriented gains sought when entering a BI project. But remember that these are not necessarily goals of the Agile process, which is more about being responsive to changing business requirements and favoring action and progress over planning and documenting.   Over time, you can see some effectiveness dividends from an Agile BI program, but appropriately setting expectations is critical.
Not Going All In
Something I and my consulting colleagues encounter frequently is the “half-steppin’” approach to Agile. For example, an organization wants to implement a waterfall pattern of requirements -> design -> development, but they want to break it in to short sprints to manage it…. Hm.   Or this one: the project team is fully versed in Agile, and has processes and tools ready to roll, but management insists on waterfall-oriented progress reporting — not available from an Agile project. The bottom line is, neither of those approaches is going to work. There’s a certain instinct to try and take the best of Column A and put it with the best of Column B. But in this case, you’re dealing with two wholly separate approaches, and mixing them successfully requires a very high level of skill and experience — if it’s possible at all.   My experience has been that if you want to do Agile, you absolutely have to jump in with both feet. The organization as a whole must be ready to embrace it, and you can’t cling to waterfall tendencies. However, that kind of full-on adoption can be easier envisioned than accomplished.
Management Doesn’t Buy It, Developers Don’t Buy It
Why is it so hard? Because people don’t like change. You need to truly assess, with clear eyes, whether your organization can change entrenched thinking to embrace Agile methods. It can be extremely difficult for business executives exposed to a career’s worth of waterfall-style projects to “get” Agile. Agile can be seen as a little bit of a dodge by these folks, especially when they don’t get the kind of estimations and progress reporting they expect. And frequently, developers aren’t sold on Agile because it usually increases their apparent workload (with adding test harnesses, extra scripting, this process, that process, JIRA, Confluence, etc…)   To avoiding managers and/or developers not buying into Agile, try to make them understand “what’s in it for them.” Managers, you will have enhanced backward-looking metrics that replace your old bogus waterfall projections. Developers, the acknowledgement that requirements are constantly changing will potentially save your sanity. Plus, with improved testing processes you will now spend less time recoding and fixing dumb errors.
You Forgot the Testing
And speaking of testing…. Failure to adopt TDD (Test-Driven Development) practices as part of implementing an Agile BI project is a huge misstep. BI testing is notoriously loosey-goosey in terms of process and tools. And with a whole team’s worth of changes rolling into source control during a sprint, you will absolutely need a structured testing methodology to ensure quality. Emphasize test-first coding to the development team in order to build unit testing not just into the process, but into the code itself.   Write QA tests based on the backlog to ensure user-facing needs are met. And don’t forget system tests — verifying that all the back-end pieces play well together, and move/change the data according to business rules.
It may seem like I’m down on Agile methods in BI, period.  But this is not the case.  As I mention above, implementing BI solutions using Agile methods can definitely provide benefits.  Chiefly, you can get product in front of consumers sooner.  Which is critical as a potential action against the truism that “people don’t know what they want until they see what they DON’T want”, because this is DEFINITELY the case when it comes to analytics and reporting… Nonetheless, I have a beef with making decisions based on buzzwords, and I can only hope that people will really consider whether Agile is a fit for their organization, project and technology before they start trying to execute.

Thoughts on “5 Common Pitfalls for Agile BI Projects”

  1. Yes!!! Thank you, Andy!!! And… Ain’t nobody got time for HALF-STEPPIN! 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Andrew Tegethoff

Andy leads Perficient's Microsoft BI team. He has 16 years of IT and software experience with a primary focus on Enterprise Information Management solutions using the Microsoft Data Platform.

More from this Author

Follow Us