Threats to the validity and value of empirical assessments of the accuracy of coverage-based fault locators

Authors:
Friedrich Steimann;Marcus Frenkel;Rui Abreu
Affiliations:
Fernuniversität in Hagen, Germany;Fernuniversität in Hagen, Germany;University of Porto, Portugal
Venue:
Proceedings of the 2013 International Symposium on Software Testing and Analysis
Year:
2013

Citing 26
Cited 2

More programming pearls: confessions of a coder

More programming pearls: confessions of a coder
Visualization of test information to assist fault localization

Proceedings of the 24th International Conference on Software Engineering
Pinpoint: Problem Determination in Large, Dynamic Internet Services

DSN '02 Proceedings of the 2002 International Conference on Dependable Systems and Networks
Locating causes of program failures

Proceedings of the 27th international conference on Software engineering
Is mutation an appropriate tool for testing experiments?

Proceedings of the 27th international conference on Software engineering
Empirical evaluation of the tarantula automatic fault-localization technique

Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering
An Evaluation of Similarity Coefficients for Software Fault Localization

PRDC '06 Proceedings of the 12th Pacific Rim International Symposium on Dependable Computing
Debugging in Parallel

Proceedings of the 2007 international symposium on Software testing and analysis
Theory-infected: or how i learned to stop worrying and love universal quantification

Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion
Semi-automatic fault localization

Semi-automatic fault localization
A Simple Coverage-Based Locator for Multiple Faults

ICST '09 Proceedings of the 2009 International Conference on Software Testing Verification and Validation
Lightweight fault-localization using multiple coverage types

ICSE '09 Proceedings of the 31st International Conference on Software Engineering
Taming coincidental correctness: Coverage refinement with context patterns to improve fault localization

ICSE '09 Proceedings of the 31st International Conference on Software Engineering
A practical evaluation of spectrum-based fault localization

Journal of Systems and Software
Evaluating the Accuracy of Fault Localization Techniques

ASE '09 Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering
Spectrum-Based Multiple Fault Localization

ASE '09 Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering
Non-parametric statistical fault localization

Journal of Systems and Software
A model for spectra-based software diagnosis

ACM Transactions on Software Engineering and Methodology (TOSEM)
Are automated debugging techniques actually helping programmers?

Proceedings of the 2011 International Symposium on Software Testing and Analysis
On the influence of multiple faults on coverage-based fault localization

Proceedings of the 2011 International Symposium on Software Testing and Analysis
The use of mutation in testing experiments and its sensitivity to external threats

Proceedings of the 2011 International Symposium on Software Testing and Analysis
Mitigating the confounding effects of program dependences for effective fault localization

Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering
Lightweight defect localization for java

ECOOP'05 Proceedings of the 19th European conference on Object-Oriented Programming
Prioritizing tests for fault localization through ambiguity group reduction

ASE '11 Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering
Sound empirical evidence in software testing

Proceedings of the 34th International Conference on Software Engineering
Reducing confounding bias in predicate-level statistical debugging metrics

Proceedings of the 34th International Conference on Software Engineering

MZoltar: automatic debugging of Android applications

Proceedings of the 2013 International Workshop on Software Development Lifecycle for Mobile
A dynamic code coverage approach to maximize fault localization efficiency

Journal of Systems and Software

Quantified Score

Hi-index	0.00

Visualization

Abstract

Resuming past work on coverage-based fault localization, we find that empirical assessments of its accuracy are subject to so many imponderables that they are of limited value. To improve on this situation, we have compiled a comprehensive list of threats to be considered when attempting such assessments in the future. In addition, we propose the establishment of theoretical lower and upper bounds of fault localization accuracy that depend on properties of the subject programs (including their test suites) only. We make a suggestion for a lower bound and show that well-known fault locators do not uniformly perform better.