On The Detection of Test Smells: A Metrics-Based Approach for General Fixture and Eager Test

• Authors: Bart Van Rompaey;Bart Du Bois;Serge Demeyer;Matthias Rieger
• Venue: IEEE Transactions on Software Engineering
• Year: 2007

As a testing method, white box testing has been demonstrated to be very efficient in early defect detection. However, white box testing introduces test co-evolution as an additional burden to software development. To mitigate the effects of co-evolution, tests should be written in a manner that makes them easy to change. Fortunately, we are able to concretely express what a good test is by exploiting the specific principles underlying white box testing. Analogous to the concept of code smells, violations of these principles are termed test smells. In this paper, we present a formal description of test smells, and propose metrics to support their detection. We validate the feasibility of detecting two test smells, General Fixture and Eager Test, by comparison with human evaluation. We demonstrate the effectiveness of the detection in the case the assessment is agreed upon by evaluators. For the General Fixture, a qualitative investigation showed that an ambiguous test smell definition prohibits the detection by metrics and suggests disentangling its definition. On the bright side, test evolvability can be more concretely expressed than general evolvability due to the exploitation of the specific principles underlying white box testing. In particular, adherence to a rigid setup-stimulate-verify-teardown cycle has been reported an essential characteristic of evolvable tests. In this article, we propose to incorporate structural characteristics of tests in the definition of test smells, thereby providing an objective means to detect test evolution obstacles. We validate the feasibility of detecting test evolution obstacles using such test smells, thereby contributing the first step to the mitigation of the cost of test co-evolution.