Differential testing: a new approach to change detection

  • Authors:
  • Robert B. Evans;Alberto Savoia

  • Affiliations:
  • Agitar Software Laboratories, Mountain View, CA;Agitar Software Laboratories, Mountain View, CA

  • Venue:
  • The 6th Joint Meeting on European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering: companion papers
  • Year:
  • 2007

Quantified Score

Hi-index 0.00

Visualization

Abstract

Regression testing, as it's commonly practiced, is unsound due to inconsistent test repair and test addition. This paper presents a new technique, differential testing, that alleviates the test repair problem and detects more changes than regression testing alone. Differential testing works by creating test suites for both the original system and the modified system and contrasting both versions of the system with these two suites. Differential testing is made possible by recent advances in automated unit test generation. Furthermore, it makes automated test generators more useful because it abstracts away the interpretation and management of large volumes of tests by focusing on the changes between test suites. In our preliminary empirical study of 3 subjects, differential testing discovered 21%, 34%, and 21% more behavior changes than regression testing alone.