What happens when you find yourself 80% through the timeline of what seems like a never ending test cycle, and can’t make sense of test statistics and defect counts? Every IT professional with enough experience has found themselves bogged down in a lengthy testing effort, buried in countless defects.
When you find yourself at this point, and the test planning failed to prepare for an elegant exit to traps like this (see previous parts in this series), it’s time to take it to the defects.
As stated in previous blog posts, in the end, Project Mangers need to only really answer the questions pertaining to real business value. Questions similar to the ones below help drive towards this:
1 – When can we go live?
2 – How much more money, time, and opportunity cost are we going to have to spend in order to go live with something worthwhile?
3 – If we went live tomorrow, what would the system look like? How would it feel? And what hindrances to core business functions would the organization experience?
When I say, “take it to the defects”, what I mean is that since the testing effort may seem like it might never end, it may be time to put all this information into perspective, and drive to completion using defects instead of test stats.
Granted, in order to do this effectively, the program should be able to run a full progression test at least once with few blocked test cases. This will enable enough data to make the defects relevant.
The first step in doing this is to make sure that all defects are classified appropriately. For example, Sev 1 should mean that the business can absolutely not function under any circumstances with this defect in existence, and there is no work around. Very rarely is this really the case. Most likely, there are many Sev 2 defects, which are the ones that would severely hinder a positive image to the program objective.
Usually, stakeholders over-dramatize the severity of certain items. The way around this issue is to ask questions like, “is it worth not going live with this defect?”. Usually, defects are more important to fix when there is no relation to timeline and budgetary factors imposed. A simple dose of tough love is usually mandated in this scenario to help identify what is truly important. Part of this activity is focused on lowering defect severity so the truly urgent items garner increased focus.
Once severity is worked out, each defect should be classified by project (if this is a larger program), by application, and by functional area (or service offering area). This last item (functional area) is very important, because it will help break down defects into manageable categories that relate directly to how the business makes money.
This prep enables the creation of a “Launch Readiness Dashboard” – see illustration. Dashboards like this help provides a business friendly picture of the current state of a given software initiative in relation to what is important to the business.
This dashboard does several things rather effectively. It represents defects across multiple projects that together, make up a complicated release. It shows these defect counts by:
1 – The greater release
2 – Applications associated with the release
3 – Individual projects
4 – And most important to the business – how they hinder the service offering that the business is trying to achieve.
This particular dashboard is presenting a testing effort that is only 40% complete (by execution stat). It’s showing a low number of defects. However, when there are hundreds of defects to contend with, this dashboard helps break the numbers down into meaningful chunks, and conveys key messages like:
- Answer questions related to what’s really important for going live.
- Color codes service offering areas that are simply not launch ready.
- Demonstrates to the business what’s still wrong with the software, and where things need to be fixed prior to launch
It’s a True Launch Readiness View of a software initiative!
Making Sense of QA Test Results – Part 3
Making Sense of QA Test Results – Part 2
Making Sense of QA Test Results – Part 1
You may also like:
- Making Sense of QA Test Results – Part 1
- Making Sense of QA Test Results – Part 3
- Making Sense of QA Test Results – Part 2
Tags: project management, qa, Quality Assurance