To run a successful testing effort requires careful and strategic preparation which should be baked in during the requirements phase. Dare I say test driven requirements? I don’t mean to preach agile or extreme programming methods, and in some cases (like large distributed multi vendor projects), waterfall delivery is inevitable. But even in the waterfall world, much can be taken from the XP. Thinking about testing as early is possible is one of these virtues.
Even if test driven requirements are not in the plan, building well thought out Business Requirements & User/Functional Scenarios is a must. Rating these items by severity (or importance to business success) is vital. Main flows and sub flows should each be rated accordingly. Test cases should then map to these specific items and they should maintain the severity level associated with them. Defects should be classified with this business severity in mind.
Mapping test cases to requirements allows for more flexibility on the reporting and defect prioritization front. You’ll be able to show test coverage to each requirement/user scenario and there will be insight into how important each test case really is. For example, a wireless testing effort could have 900 test cases, with an average of 10 conditions per case, which adds up to some pretty extensive execution time and all the overhead that results from that. However, identifying which tests cases MUST pass is only possible by rating them by business priority or severity.
So what happens when there is either no time to do all this test prep and planning, or the test effort is already underway, and this planning did not occur?
The answer is if the test effort is already under way, it is often too late to retrofit it. What usually happens it that testing gets bogged down and delayed through blocked test cases or defect resolution delays. Time runs out and questions come up about what to do to bridge the gap.
For example, you’ll be 70% through test execution with a 60% pass rate. Trending (a topic for a later discussion) shows that the effort will take 10 more weeks, when only 2 remain in the budget/timeline.
Without prioritized test cases, it’s hard to tell what cases absolutely need to be run. Maybe another approach is warranted at this point.
The reality is that unless there is a way to weed out lower priority test cases, each one needs to be run at least once. If there is time to thin out the noise, then it behooves the entire effort to exercise on this analysis.
But in the heat of delivery, even this analysis may not produce the most productive approach. When you’ve reached this point, sometimes the only course of action is to take a careful look at the defects – analyzing what items really need to be resolved before going live and which ones don’t.
Note: This author advocates that in most software development projects, it makes little sense, and may very well impede the entire business objective, to set a goal of fixing all defects. Only high priority defects need to be resolved when the cost of fixing more means more budget and more time that the application could be live, making money, or increasing efficiency.
So essentially, this advocates paying more attention to what’s wrong with the system (through defects), than how what the “Pass Rate” is of hundreds of little understood test cases. The difference between gauging completion through test case execution vs. defect resolution may seem minute, but there is a huge difference when looked at more closely. Without the careful planning of a well run test effort (and most are not well planned), the entire program could be at the mercy of the test cases that were never properly reviewed by the people that count the most. There could be too little coverage, or in most cases, far too much. And defects that arise from the effort often lack relevance to true business objectives (even when the business reviews the defects). Back to questions like:
1 – Can we launch the new service?
2 – Will it provide profitable benefit to our customers – with a reasonable (but maybe not perfect) user experience?
3 – Can we operate more efficiently with this new software?
So when things are running late and the testing cycle seems endless, it means it’s time to start looking at defects. The goal now is to minimize how many defects need to be resolved in order to go live. The premise is that there could be 200 defects, but only 20 are really needed to launch, and another 50 are needed soon after launch volume orders. These are arbitrary numbers, but are often true. This is not to say that there is benefit in launching something that hurts brand image, but there is often a careful balance of where it becomes too costly to wait to fix certain items.
A great example: You’re a wireless carrier coming out with a new form of service. One of the plans offers unlimited data. There are a series of defects related to how usage and billing is presented to the customer. One of the defects, involves a MB data used bar on a GUI screen, that always shows “<1%” on plans that have unlimited data usage. This might confuse the customer, however common sense would tell them that since the bar never moves past 1% of usage, they have no reason for concern. This is probably a defect, and at some point should be fixed, but during crunch time, when there are plenty of sev 1 or 2 defects to content with, there is little time to spend on resolving such issues.
However, it takes a touch of tough love to come to this conclusion with all stakeholders.
The next part in this blog series will discuss how to present the overall health of a test effort from a defect perspective.
Making Sense of QA Test Results – Part 2
Making Sense of QA Test Results – Part 1
[...] Making Sense of QA Test Results – Part 3 [...]