Feedback driven adaptive combinatorial testing

  • Authors:

  • Emine Dumlu;Cemal Yilmaz;Myra B. Cohen;Adam Porter

  • Affiliations:

  • Sabanci University, Istanbul, Turkey;Sabanci University, Istanbul, Turkey;University of Nebraska-Lincoln, Lincoln, NE;University of Maryland, College Park, MD

  • Venue:
  • Proceedings of the 2011 International Symposium on Software Testing and Analysis
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

The configuration spaces of modern software systems are too large to test exhaustively. Combinatorial interaction testing (CIT) approaches, such as covering arrays, systematically sample the configuration space and test only the selected configurations. The basic justification for CIT approaches is that they can cost-effectively exercise all system behaviors caused by the settings of t or fewer options. We conjecture, however, that in practice many such behaviors are not actually tested because of masking effects -- failures that perturb execution so as to prevent some behaviors from being exercised. In this work we present a feedback-driven, adaptive, combinatorial testing approach aimed at detecting and working around masking effects. At each iteration we detect potential masking effects, heuristically isolate their likely causes, and then generate new covering arrays that allow previously masked combinations to be tested in the subsequent iteration. We empirically assess the effectiveness of the proposed approach on two large widely used open source software systems. Our results suggest that masking effects do exist and that our approach provides a promising and efficient way to work around them.