The Moment We've All Been Waiting for...NVP's "Making ROI the Easy Part" Report

Written by: Neil Price-Jones, CSTE - NVP Software Testing

It is easy to find a number of papers on the ROI of testing. Most of them are published by Test Tool Vendors and show quick payback, fast breakeven, and substantial ROI values that justify the expenditure on automation within months, or at the most a couple of years. The calculations are based on purchase of the tool and expected increases in efficiency and effectiveness of testing using the vendor's automation methodology. We do not dispute that there are gains to be made from automation and that some of the calculations make good arguments for investing in the correct technology (testing tool) for your organization and implementing it. However, many companies are not in a position to buy a tool, no matter how seductive the argument. Many companies cannot buy a tool at this time for many reasons including, the current economic climate, a deflated testing group, and other cost cutting concerns. We will be lucky to retain our own testing group without cutbacks! Now, more than ever, testing groups will have to justify their existence. In order to do that, hard figures and a formula to calculate the value of what we do, are necessary.

The following spreadsheet will help you to justify your work to upper management in the terms that they require. It is flexible and you can fill in your own figures to determine how your team fits into the Profit and Loss for the organization. In order to do this, you will need some historical information or you will have to make some estimates and revise them later. Be prepared to defend your figures.
We welcome feedback and corrections.

Definitions and Initial Set-Up

Manual Testing Productivity: This is the rate at which you or your testers can execute tests currently. In order to calculate this, you will require information on every run (defined as a single execution of a testcase) that has occurred. It is important to realize that this can vary from minutes (for a simple GUI dialogue test) to hours and days (for performance tests). An average is required. This is sometimes referred to as Tester Efficiency.

Manual Testing Effectiveness: This is a measure of how many defects are found before the next phase in product development and ultimately before the product is released to the final users. This can be measured at each Phase or Stage and used for Quality Improvements or it can be calculated as the ratio of defects that escaped to production versus the sum of all the defects found in the all the phases prior to release to the final user.

Losses Due to Late Delivery: Whether the product is sold to a final consumer or whether it is used internally in support of your operations or business, there is a cost to late delivery. It may be in make up internal work, lost customers, or reduced revenue stream. An estimate of this is necessary in order to determine the benefit of testing.

The Effects of Escaped Bugs

Following are just some of the negative effects of escaped bugs. This list does not encompass every possible risk but does summarize some very possible and likely scenarios.

Unexpected expenses arise when a bug-fix release must be issued or a patch or service pack must be sent out.

Loss of revenue and increased costs due to interruptions or halting of revenue-generating production systems.

Increased customer support expenses - more frequent support calls lead to a need for more staff, overhead costs, increased supplies, salaries, etc.

Loss of customers to competitors as a result of undesirable customer experience.

Loss of potential customers to competitors due to negative company image, negative word-of-mouth - overall result = lost revenue.

Financial losses due to incorrect, missing, or late information.

Losses due to disruption or outage of a process or piece of equipment that other systems depend on (spoilage losses, opportunity costs, higher indirect and overhead costs).

Increased interest or insurance costs due to instability resulting in an unfavourable risk assessment.

Lawsuits for injury, death, or damage resulting from software failure.

Risk

In order to apply the model to any testing we complete, we need to calculate the risk involved in the product release. The list in the previous section provides some of the deleterious effects of released bugs. We need to take each one of those and assign a Risk Factor to it.

First, however, we have to acknowledge the myth of complete testing. There is no possibility of manual testing covering all the possible combinations in a program and as programs have grown so has the impossibility of testing even a small subset of the possibilities. We have to use Risk to cut this to a manageable number.

Risk is defined as:
RISK = Probability X Impact

Probability can be defined on a numeric scale or H, M, L (High, Medium, Low)

Impact can be scaled in a similar way or it can be calculated based on financial impact ranging from little impact to a company wide loss and shutdown.[1]

Once we have determined the areas of High, Medium and Low risk we can map these back to the testcases and designate them as High, Medium and Low Priority according to how much they address the Risk.

Discouraging Figures

There are three sources of discouraging figures in the testing world.

1. Some quoted studies indicate that there is an average of 6 defects per thousand lines of code (method of counting lines being in dispute). However, in a 2 million line system that means 12,000 defects.

2. The next one is the break/fix ratio. This is the ratio of new bugs introduced in an effort to fix the existing ones. Some programs have very high break/fix ratios (a lot of new bugs are introduced in the course of fixing already found bugs) while other programs fix once and do not break anything else giving a ratio of zero. The worse the break/fix ratio, the more iterations of testing will be required to drive the defect count to an acceptable level. Studies have indicated ratios as high as 1/6 (i.e. 1 error is introduced for every 6 fixed).

3. Lastly, the notion of tester error (one testers are prone to deny). These are testcases, test inputs, and expected results that are simply wrong. They lead to spurious reported errors and wasted time on all parts.

[1] Note that this does not apply to medical or flight control products which can kill people and have a completely different scaling.

This report then moves into how to begin the calculations (starting with necessary assumptions). We are unable to include the spreadsheet via the Bug Report but we are hoping to have the report and accompanying spreadsheet available on the website shortly. In the meantime, if you would like a copy of the spreadsheet, please email nicole.borsoi@nvp-inc.com or reply to this email with your request. Please allow a 5 business day grace period due to business travel etc. Thank you for your interest in our report.