Faults on its sleeve: amplifying software reliability testing

• Authors: Dick Hamlet;Jeff Voas
• Venue: ISSTA '93 Proceedings of the 1993 ACM SIGSOFT international symposium on Software testing and analysis
• Year: 1993

Most of the effort that goes into improving the quality of software paradoxically does not lead to quantitative, measurable quality. Software developers and quality-assurance organizations spend a great deal of effort preventing, detecting, and removing “defects”—parts of software responsible for operational failure. But software quality can be measured only by statistical parameters like hazard rate and mean time to failure, measures whose connection with defects and with the development process is little understood.At the same time, direct reliability assessment by random testing of software is impractical. The levels we would like to achieve, on the order of 106 - 108 executions without failure, cannot be established in reasonable time. Some limitations of reliability testing can be overcome but the “ultrareliable” region above 108 failure-free executions is likely to remain forever untestable.We propose a new way of looking at the software reliability program. Defect-based efforts should amplify the significance of reliability testing. That is, developers should demonstrate that the actual reliability is better than the measurement. We give an example of a simple reliability-amplification technique, and suggest applications to systematic testing and formal development methods.