Data Diversity: An Approach to Software Fault Tolerance
IEEE Transactions on Computers - Fault-Tolerant Computing
Genetic Algorithms in Search, Optimization and Machine Learning
Genetic Algorithms in Search, Optimization and Machine Learning
Art of Software Testing
Introduction to Algorithms
DART: directed automated random testing
Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation
MuJava: an automated class mutation system: Research Articles
Software Testing, Verification & Reliability
Lattice-based adaptive random testing
Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering
An empirical analysis and comparison of random testing techniques
Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering
Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria
IEEE Transactions on Software Engineering
Feedback-Directed Random Test Generation
ICSE '07 Proceedings of the 29th international conference on Software Engineering
Search Algorithms for Regression Test Case Prioritization
IEEE Transactions on Software Engineering
Towards the determination of typical failure patterns
Fourth international workshop on Software quality assurance: in conjunction with the 6th ESEC/FSE joint meeting
A Domain Strategy for Computer Program Testing
IEEE Transactions on Software Engineering
An upper bound on software testing effectiveness
ACM Transactions on Software Engineering and Methodology (TOSEM)
ARTOO: adaptive random testing for object-oriented software
Proceedings of the 30th international conference on Software engineering
Antirandom testing: a distance-based approach
VLSI Design
Numerical Recipes 3rd Edition: The Art of Scientific Computing
Numerical Recipes 3rd Edition: The Art of Scientific Computing
Distributing test cases more evenly in adaptive random testing
Journal of Systems and Software
Enhanced lattice-based adaptive random testing
Proceedings of the 2009 ACM symposium on Applied Computing
Adaptive random testing based on distribution metrics
Journal of Systems and Software
Adaptive Random Testing: The ART of test case diversity
Journal of Systems and Software
An Indepth Study of Mirror Adaptive Random Testing
QSIC '09 Proceedings of the 2009 Ninth International Conference on Quality Software
A Divergence-Oriented Approach to Adaptive Random Testing of Java Programs
ASE '09 Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering
Adaptive Random Test Case Prioritization
ASE '09 Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering
Formal analysis of the effectiveness and predictability of random testing
Proceedings of the 19th international symposium on Software testing and analysis
Using Coverage Information to Guide Test Case Selection in Adaptive Random Testing
COMPSACW '10 Proceedings of the 2010 IEEE 34th Annual Computer Software and Applications Conference Workshops
Normalized Adaptive Random Test for Integration Tests
COMPSACW '10 Proceedings of the 2010 IEEE 34th Annual Computer Software and Applications Conference Workshops
Reducing the cost of model-based testing through test case diversity
ICTSS'10 Proceedings of the 22nd IFIP WG 6.1 international conference on Testing software and systems
Black-box system testing of real-time embedded systems using random and search-based testing
ICTSS'10 Proceedings of the 22nd IFIP WG 6.1 international conference on Testing software and systems
Proceedings of the 33rd International Conference on Software Engineering
ASIAN'04 Proceedings of the 9th Asian Computing Science conference on Advances in Computer Science: dedicated to Jean-Louis Lassez on the Occasion of His 5th Cycle Birthday
On the danger of coverage directed test case generation
FASE'12 Proceedings of the 15th international conference on Fundamental Approaches to Software Engineering
Proceedings of the 2012 International Symposium on Software Testing and Analysis
Diversity maximization speedup for fault localization
Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering
CarFast: achieving higher statement coverage faster
Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering
An orchestrated survey of methodologies for automated software test case generation
Journal of Systems and Software
Hi-index | 0.01 |
Adaptive Random Testing (ART) has been proposed as an enhancement to random testing, based on assumptions on how failing test cases are distributed in the input domain. The main assumption is that failing test cases are usually grouped into contiguous regions. Many papers have been published in which ART has been described as an effective alternative to random testing when using the average number of test case executions needed to find a failure (F-measure). But all the work in the literature is based either on simulations or case studies with unreasonably high failure rates. In this paper, we report on the largest empirical analysis of ART in the literature, in which 3727 mutated programs and nearly ten trillion test cases were used. Results show that ART is highly inefficient even on trivial problems when accounting for distance calculations among test cases, to an extent that probably prevents its practical use in most situations. For example, on the infamous Triangle Classification program, random testing finds failures in few milliseconds whereas ART execution time is prohibitive. Even when assuming a small, fixed size test set and looking at the probability of failure (P-measure), ART only fares slightly better than random testing, which is not sufficient to make it applicable in realistic conditions. We provide precise explanations of this phenomenon based on rigorous empirical analyses. For the simpler case of single-dimension input domains, we also perform formal analyses to support our claim that ART is of little use in most situations, unless drastic enhancements are developed. Such analyses help us explain some of the empirical results and identify the components of ART that need to be improved to make it a viable option in practice.