Using coverage effectiveness to evaluate test suite prioritizations

  • Authors:
  • Gregory M. Kapfhammer;Mary Lou Soffa

  • Affiliations:
  • Allegheny College;University of Virginia

  • Venue:
  • Proceedings of the 1st ACM international workshop on Empirical assessment of software engineering languages and technologies: held in conjunction with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE) 2007
  • Year:
  • 2007

Quantified Score

Hi-index 0.00

Visualization

Abstract

Regression test suite prioritization techniques reorder a test suite with the goal of ensuring that the reorganized test suite finds faults faster than the initial ordering. It is challenging to empirically evaluate the effectiveness of a new test case arrangement because existing metrics (i) require fault seeding or (ii) ignore test case costs. This paper presents a coverage effectiveness (CE) metric that (i) obviates the need to seed faults into the program under test and (ii) incorporates available data about test case execution times. A test suite is awarded a high CE value when it quickly covers the test requirements. It is possible to calculate coverage effectiveness regardless of the coverage criterion that is chosen to evaluate test case quality. The availability of an open source CE calculator enables future case studies and controlled experiments to use coverage effectiveness when evaluating different approaches to test suite prioritization.