Large software programs eventually involve equally large and cumbersome testing components.  A great deal of work goes into planning, organizing, executing, and reporting on these testing efforts.  Furthermore, most development initiatives run way past original timelines and naturally try to squeeze the testing initiative into slimmer timeframes.

What comes out of all of this usually amounts to a series of raw statistics, morphed into trend lines, bar graphs, limitless numbers, and complex dashboards that few people understand.  If you’re lucky, there might be someone that can show a neat test coverage chart that shows how many test cases map to different parts of the application space, and how many of these test cases pass and fail.

It usually takes a great deal of effort, and in some cases, a lot of money in testing software tools to put all of this information together.  But hidden within all of this information (vast rows of statistics and graphs), is a little secret that few people recognize…  It doesn’t mean anything.

How often have you seen a report that says:

Total Test Cases: 2000

Test Case Execution %:  70%

Test Case Pass %: 72%

Presenting this to stakeholders does very little good.  As a side note, showing defect counts probably does more harm than good.  In some situations, a program could have a 65% pass rate or 400 defects and still be launch ready, and make money, or increase efficiency with the offering.   Showing these types of statistics requires the audience to have a deep understanding of the nature and quality of the test effort.  It requires them to know whether all test cases need to pass or if all defect resolutions are required for launch.

So this brings us to a really important point.  All stakeholders really want to know is:

“How much more time and money is it going to take to launch something worthwhile”.

They won’t find the answer to this question in these statistics.

Let’s say the initiative is a new wireless service offering.  What the stakeholders want to know are things like:

1. Can we advertise for this service on our website.
2. Should I buy add space in magazines and commercial time (a very time sensitive objective).
3. Can we activate customers on this new service?  Or can I activate 95% of them without too much pain.
4. Can we track their usage?
5. Can we bill them for the services we are provided?
6. Can we support my customers effectively when they are having problems?
7. Can we transfer, suspend, restore, and disconnect the service?
8. Can we do these things while providing a reasonable customer experience.

If these are the questions that stakeholders need answered, then the entire testing effort should be focused on providing answers to these types of questions.

From a test design perspective, there are certain things that should be built into the test cases from the conception.  All test cases and scenarios should be mapped to how they affect the service offering (in sensible and high level ways).   Categorizing test cases by service offering area is an effective way to do this.  Each test case should be categorized by severity as well, where the highest severity tests should be the ones that the business can’t launch without.  Second tier test cases are the ones the business feels they can’t live without, but when push comes to shove, the probably could.  And lower severity levels follow a logical path in reference to launch readiness.

Baking in this kind of careful preparation in the test design and planning process enables  far more options when it’s time to report on the test results.  Instead of showing raw statistics of test case counts, execution, blocked, pass, & fail counts, which again mean very little, you could show a diagram of the service offering, in terms of business value add, and map critical, and tier 2 test case stats to each critical business area.  The key is to show as few stats as possible while projecting a clear message of the health of the offering.

Where this takes us is to a “Launch Readiness Dashboard”, which shows the health of the software offering based on thoughtfully designed test cases, mapping to real business value.  The key is to make a big deal out of the test cases that have the most business impact, and putting the edge scenarios into a more relevant perspective.   Often this is not done, which causes stakeholders to say things like:

“We can’t have any defects”

“90% of test cases have to pass”

Neither milestone is beneficial from a business case perspective.

So what happens when there is either no time to do all this test prep and planning, or the test effort is already underway, and this planning did not occur?  An approach to this very common situation will be discussed in Part 3 of this blog series.

Making Sense of QA Test Results – Part 3

Making Sense of QA Test Results – Part 1

You may also like:

• Making Sense of QA Test Results – Part 1

This entry was posted on Friday, February 5th, 2010 at 1:25 pm and is filed under News. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.