Monday, July 13, 2009

When Am I Done Testing?

A rough estimate puts the costs of software-related downtime at about US$300 billion annually. Consider these examples:
• In September 2007, a computer system in Atlanta that processes pilots’ flight plans and sends them to air traffic controllers fails. The agency reroutes the system’s functions to another computer in Salt Lake City, which overloads because of the increased volume of data, magnifying the problem.
• A telecommunications provider spends US$3 million on software application • support over six months instead of on new application development.
• A health insurance company loses US$20 billion annually in lost business • and repair work related to production performance problems.

Research and experience have shown that the cost of finding and removing defects grows exponentially with the quality of the code. That is, as each defect is removed, the cost of finding and removing the next defect increases. Hence, it is a practical impossibility to deliver zero-defect software. C-level executives understandably want products whose defects have a minimal impact on users—either functionally or financially. But such certainty—and the enhanced ability to avoid disasters like those described above—costs money. Hundreds of millions of dollars are spent yearly on software testing. Indeed, IBM’s direct experience with organizations worldwide and data gathered by researchers working with hundreds of companies show that most firms invest 25 percent or more of their development lifecycle time and cost in quality assurance.

A key point in any development process, therefore, is the point at which testing ends and the organization moves ahead with deployment. This is the time in the life of every development project when the program manager has to ask the very practical questions:
• “Is it wise to continue spending money on quality assurance?”
• “Will further testing cost more than it is worth?”
• “Is the software ready for release?”
• “How do we know when we’re done testing?”

This white paper provides an innovative framework for answering those questions—a new barometer for measuring the business risk of release versus the cost of continued testing.

Download the Entire White Paper from IBM