Software Testability: The New Verification

• Authors: Jeffrey M. Voas;Keith W. Miller
• Venue: IEEE Software
• Year: 1995

Software verification is often the last defense against disasters caused by faulty software development. When lives and fortunes depend on software, software quality and its verification demand increased attention. As software begins to replace human decision-makers, a fundamental concern is whether a machine will be able to perform the tasks with the same level of precision as a skilled person. If not, a catastrophe may be caused by an automated system that is less reliable than a manual one. How do you assess that critical automated systems are acceptably safe and reliable? In this article, we present a new view of verification and offer techniques that will help developers make this assessment. Our view, which we label software testability, is somewhat more inclusive than traditional verification and testability views Every system has a true (or fixed) reliability that is generally unknown. Software testability, software testing, and formal verification are three pieces of the reliability puzzle, which developers must complete to get a picture of the software's true reliability. Each of the three puzzle pieces offers a unique bit of information about software quality. The goal is to get all three. Testability information cannot replace testing and formal verification; but neither should developers rely exclusively on testing and formal verification. To have highly reliable software, the software must have high testability, undergone enormous amounts of successful testing, and experienced formal verification.Our focus here is on one part of the puzzle, testability - how to design for it and how to measure the degree to which you have achieved it. To better illustrate what we mean by software testability, consider a simple analogy that shows how testability can enhance testing. Suppose software faults were gold. Software testing would be the actual mining process,. and software testability would be the geologist's survey before mining begins. The geologist does not actually dig for the gold, but rather establishes the likelihood that digging at a particular spot would be rewarding. At one location, the geologist might say, "This valley may or may not have gold, but if it does, it will be in the top 50 feet and all over the valley." At another location, the geologist might say, "If you don't find gold in the first 10 feet on this plateau, there is no gold. However, on the next plateau you will have to dig 100 feet before you can be sure there is no gold." We are particularly interested in designing software to increase its testability. Although the existence of an upper bound on testability is solely our conjecture, our research using sensitivity analysis and studying software's tendency to not reveal faults during testing suggests that such exists. We challenge software testing researchers to consider this it. A given piece of software will or will not hide a given fault from testing. We have found that it is possible to examine this code characteristic - software testability - without knowing if a particular fault exists in that software, and without reference to correctness. Because it does not rely on correctness, software testability gives a new perspective on code development.