Search-based system testing: high coverage, no false alarms

  • Authors:
  • Florian Gross;Gordon Fraser;Andreas Zeller

  • Affiliations:
  • Saarland University, Germany;Saarland University, Germany;Saarland University, Germany

  • Venue:
  • Proceedings of the 2012 International Symposium on Software Testing and Analysis
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

Modern test case generation techniques can automatically achieve high code coverage. If they operate on the unit level, they run the risk of generating inputs infeasible in reality, which, when causing failures, are painful to identify and eliminate. Running a unit test generator on five open source Java programs, we found that all of the 181 reported failures were false failures—that is, indicating a problem in the generated test case rather than the program. By generating test cases at the GUI level, our EXSYST prototype can avoid such false alarms by construction. In our evaluation, it achieves higher coverage than search-based test generators at the unit level; yet, every failure can be shown to be caused by a real sequence of input events. Whenever a system interface is available, we recommend considering search-based system testing as an alternative to avoid false failures.