Effect of test set minimization on fault detection effectiveness
Proceedings of the 17th international conference on Software engineering
An experimental evaluation of selective mutation
ICSE '93 Proceedings of the 15th international conference on Software Engineering
Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria
ICSE '94 Proceedings of the 16th international conference on Software engineering
Further empirical studies of test effectiveness
SIGSOFT '98/FSE-6 Proceedings of the 6th ACM SIGSOFT international symposium on Foundations of software engineering
Proceedings of the ACM SIGPLAN 1999 conference on Programming language design and implementation
Automatically validating temporal safety properties of interfaces
SPIN '01 Proceedings of the 8th international SPIN workshop on Model checking of software
POPL '02 Proceedings of the 29th ACM SIGPLAN-SIGACT symposium on Principles of programming languages
Prioritizing Test Cases For Regression Testing
IEEE Transactions on Software Engineering
An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing
IEEE Transactions on Software Engineering
Improving test suites via operational abstraction
Proceedings of the 25th International Conference on Software Engineering
Empirical Evaluation of the Textual Differencing Regression Testing Technique
ICSM '98 Proceedings of the International Conference on Software Maintenance
Explaining abstract counterexamples
Proceedings of the 12th ACM SIGSOFT twelfth international symposium on Foundations of software engineering
Is mutation an appropriate tool for testing experiments?
Proceedings of the 27th international conference on Software engineering
The effect of code coverage on fault detection under different testing profiles
A-MOST '05 Proceedings of the 1st international workshop on Advances in model-based testing
Empirical Software Engineering
Automated path generation for software fault localization
Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering
Test input generation for java containers using state matching
Proceedings of the 2006 international symposium on Software testing and analysis
Compositional dynamic test generation
Proceedings of the 34th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages
Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria
IEEE Transactions on Software Engineering
Feedback-Directed Random Test Generation
ICSE '07 Proceedings of the 29th international conference on Software Engineering
Randomized Differential Testing as a Prelude to Formal Verification
ICSE '07 Proceedings of the 29th international conference on Software Engineering
Testing Programs with the Aid of a Compiler
IEEE Transactions on Software Engineering
International Journal on Software Tools for Technology Transfer (STTT)
Introduction to Software Testing
Introduction to Software Testing
Sufficient mutation operators for measuring test effectiveness
Proceedings of the 30th international conference on Software engineering
HOLMES: Effective statistical debugging via efficient path profiling
ICSE '09 Proceedings of the 31st International Conference on Software Engineering
The influence of size and coverage on test suite effectiveness
Proceedings of the eighteenth international symposium on Software testing and analysis
Javalanche: efficient mutation testing for Java
Proceedings of the the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering
Introduction to Algorithms, Third Edition
Introduction to Algorithms, Third Edition
A Comparison of Constraint-Based and Sequence-Based Generation of Complex Input Data Structures
ICSTW '10 Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops
Is operator-based mutant selection superior to random mutant selection?
Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1
Analysis of invariants for efficient bounded verification
Proceedings of the 19th international symposium on Software testing and analysis
Proceedings of the 33rd International Conference on Software Engineering
Testing container classes: random or systematic?
FASE'11/ETAPS'11 Proceedings of the 14th international conference on Fundamental approaches to software engineering: part of the joint European conferences on theory and practice of software
An Analysis and Survey of the Development of Mutation Testing
IEEE Transactions on Software Engineering
A theory of predicate-complete test coverage and generation
FMCO'04 Proceedings of the Third international conference on Formal Methods for Components and Objects
(Quickly) testing the tester via path coverage
WODA '09 Proceedings of the Seventh International Workshop on Dynamic Analysis
A simple generic library for c
ICSR'06 Proceedings of the 9th international conference on Reuse of Off-the-Shelf Components
Coverage rewarded: Test input generation via adaptation-based programming
ASE '11 Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering
Proceedings of the 2012 International Symposium on Software Testing and Analysis
Comparing multi-point stride coverage and dataflow coverage
Proceedings of the 2013 International Conference on Software Engineering
Hi-index | 0.00 |
A fundamental question in software testing research is how to compare test suites, often as a means for comparing test-generation techniques. Researchers frequently compare test suites by measuring their coverage. A coverage criterion C provides a set of test requirements and measures how many requirements a given suite satisfies. A suite that satisfies 100% of the (feasible) requirements is C-adequate. Previous rigorous evaluations of coverage criteria mostly focused on such adequate test suites: given criteria C and C′, are C-adequate suites (on average) more effective than C′-adequate suites? However, in many realistic cases producing adequate suites is impractical or even impossible. We present the first extensive study that evaluates coverage criteria for the common case of non-adequate test suites: given criteria C and C′, which one is better to use to compare test suites? Namely, if suites T1, T2 . . . Tn have coverage values c1, c2 . . . cn for C and c′1, c′2 . . . c′n for C′, is it better to compare suites based on c1, c2 . . . cn or based on c′1, c′ 2 . . . c′n? We evaluate a large set of plausible criteria, including statement and branch coverage, as well as stronger criteria used in recent studies. Two criteria perform best: branch coverage and an intra-procedural acyclic path coverage.