Test Case Prioritization: A Family of Empirical Studies
IEEE Transactions on Software Engineering
Effectively prioritizing tests in development environment
ISSTA '02 Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis
Visualization of test information to assist fault localization
Proceedings of the 24th International Conference on Software Engineering
Test Case Prioritization: An Empirical Study
ICSM '99 Proceedings of the IEEE International Conference on Software Maintenance
A Study of Effective Regression Testing in Practice
ISSRE '97 Proceedings of the Eighth International Symposium on Software Reliability Engineering
Selecting a Cost-Effective Test Case Prioritization Technique
Software Quality Control
Locating causes of program failures
Proceedings of the 27th international conference on Software engineering
Proceedings of the 27th international conference on Software engineering
Scalable statistical bug isolation
Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation
SOBER: statistical model-based bug localization
Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering
Empirical evaluation of the tarantula automatic fault-localization technique
Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering
Improving test suites for efficient fault localization
Proceedings of the 28th international conference on Software engineering
Object distance and its application to adaptive random testing of object-oriented programs
Proceedings of the 1st international workshop on Random testing
TimeAware test suite prioritization
Proceedings of the 2006 international symposium on Software testing and analysis
Continuous Integration: Improving Software Quality and Reducing Risk (The Addison-Wesley Signature Series)
On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques
IEEE Transactions on Software Engineering
Search Algorithms for Regression Test Case Prioritization
IEEE Transactions on Software Engineering
Experimental assessment of random testing for object-oriented software
Proceedings of the 2007 international symposium on Software testing and analysis
An upper bound on software testing effectiveness
ACM Transactions on Software Engineering and Methodology (TOSEM)
ARTOO: adaptive random testing for object-oriented software
Proceedings of the 30th international conference on Software engineering
An empirical study of the effects of test-suite reduction on fault localization
Proceedings of the 30th international conference on Software engineering
Fault localization using value replacement
ISSTA '08 Proceedings of the 2008 international symposium on Software testing and analysis
Time-aware test-case prioritization using integer linear programming
Proceedings of the eighteenth international symposium on Software testing and analysis
Recomputing Coverage Information to Assist Regression Testing
IEEE Transactions on Software Engineering
Capturing propagation of infected program states
Proceedings of the the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering
Is non-parametric hypothesis testing model robust for statistical fault localization?
Information and Software Technology
A practical evaluation of spectrum-based fault localization
Journal of Systems and Software
How Well Do Test Case Prioritization Techniques Support Statistical Fault Localization
COMPSAC '09 Proceedings of the 2009 33rd Annual IEEE International Computer Software and Applications Conference - Volume 01
Adaptive Random Testing: The ART of test case diversity
Journal of Systems and Software
Fault localization through evaluation sequences
Journal of Systems and Software
A family of code coverage-based heuristics for effective fault localization
Journal of Systems and Software
Spectrum-Based Multiple Fault Localization
ASE '09 Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering
Adaptive Random Test Case Prioritization
ASE '09 Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering
A Dynamic Test Cluster Sampling Strategy by Leveraging Execution Spectra Information
ICST '10 Proceedings of the 2010 Third International Conference on Software Testing, Verification and Validation
Prioritizing Tests for Software Fault Localization
QSIC '10 Proceedings of the 2010 10th International Conference on Quality Software
Regression Testing Process Improvement for Specification Evolution of Real-World Protocol Software
QSIC '10 Proceedings of the 2010 10th International Conference on Quality Software
A Simulation Study on Some Search Algorithms for Regression Test Case Prioritization
QSIC '10 Proceedings of the 2010 10th International Conference on Quality Software
An Improved Regression Test Selection Technique by Clustering Execution Profiles
QSIC '10 Proceedings of the 2010 10th International Conference on Quality Software
On the Integration of Test Adequacy, Test Case Prioritization, and Statistical Fault Localization
QSIC '10 Proceedings of the 2010 10th International Conference on Quality Software
KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs
OSDI'08 Proceedings of the 8th USENIX conference on Operating systems design and implementation
Using Coverage Information to Guide Test Case Selection in Adaptive Random Testing
COMPSACW '10 Proceedings of the 2010 IEEE 34th Annual Computer Software and Applications Conference Workshops
Non-parametric statistical fault localization
Journal of Systems and Software
An empirical study on the effectiveness of time-aware test case prioritization techniques
Proceedings of the 2011 ACM Symposium on Applied Computing
A model for spectra-based software diagnosis
ACM Transactions on Software Engineering and Methodology (TOSEM)
A general noise-reduction framework for fault localization of Java programs
Information and Software Technology
Information and Software Technology
Mutation-oriented test data augmentation for GUI software fault localization
Information and Software Technology
Hi-index | 0.00 |
Context: Effective test case prioritization shortens the time to detect failures, and yet the use of fewer test cases may compromise the effectiveness of subsequent fault localization. Objective: The paper aims at finding whether several previously identified effectiveness factors of test case prioritization techniques, namely strategy, coverage granularity, and time cost, have observable consequences on the effectiveness of statistical fault localization techniques. Method: This paper uses a controlled experiment to examine these factors. The experiment includes 16 test case prioritization techniques and four statistical fault localization techniques using the Siemens suite of programs as well as grep, gzip, sed, and flex as subjects. The experiment studies the effects of the percentage of code examined to locate faults from these benchmark subjects after a given number of failures have been observed. Results: We find that if testers have a budgetary concern on the number of test cases for regression testing, the use of test case prioritization can save up to 40% of test case executions for commit builds without significantly affecting the effectiveness of fault localization. A statistical fault localization technique using a smaller fraction of a prioritized test suite is found to compromise its effectiveness seriously. Despite the presence of some variations, the inclusion of more failed test cases will generally improve the fault localization effectiveness during the integration process. Interestingly, during the variation periods, adding more failed test cases actually deteriorates the fault localization effectiveness. In terms of strategies, Random is found to be the most effective, followed by the ART and Additional strategies, while the Total strategy is the least effective. We do not observe sufficient empirical evidence to conclude that using different coverage granularity levels have different overall effects. Conclusion: The paper empirically identifies that strategy and time-cost of test case prioritization techniques are key factors affecting the effectiveness of statistical fault localization, while coverage granularity is not a significant factor. It also identifies a mid-range deterioration in fault localization effectiveness when adding more test cases to facilitate debugging.