A practical evaluation of spectrum-based fault localization

  • Authors:

  • Rui Abreu;Peter Zoeteweij;Rob Golsteijn;Arjan J. C. van Gemund

  • Affiliations:

  • Delft University of Technology, The Netherlands;Delft University of Technology, The Netherlands;NXP Semiconductors, The Netherlands;Delft University of Technology, The Netherlands

  • Venue:
  • Journal of Systems and Software
  • Year:
  • 2009

Quantified Score

Hi-index 0.00

Visualization

Abstract

Spectrum-based fault localization (SFL) shortens the test-diagnose-repair cycle by reducing the debugging effort. As a light-weight automated diagnosis technique it can easily be integrated with existing testing schemes. Since SFL is based on discovering statistical coincidences between system failures and the activity of the different parts of a system, its diagnostic accuracy is inherently limited. Using a common benchmark consisting of the Siemens set and the space program, we investigate this diagnostic accuracy as a function of several parameters (such as quality and quantity of the program spectra collected during the execution of the system), some of which directly relate to test design. Our results indicate that the superior performance of a particular similarity coefficient, used to analyze the program spectra, is largely independent of test design. Furthermore, near-optimal diagnostic accuracy (exonerating over 80% of the blocks of code on average) is already obtained for low-quality error observations and limited numbers of test cases. In addition to establishing these results in the controlled environment of our benchmark set, we show that SFL can effectively be applied in the context of embedded software development in an industrial environment.