Orthogonal Latin squares: an application of experiment design to compiler testing
Communications of the ACM
Applying design of experiments to software testing: experience report
ICSE '97 Proceedings of the 19th international conference on Software engineering
The AETG System: An Approach to Testing Based on Combinatorial Design
IEEE Transactions on Software Engineering
Model-based testing in practice
Proceedings of the 21st international conference on Software engineering
Data mining: practical machine learning tools and techniques with Java implementations
Data mining: practical machine learning tools and techniques with Java implementations
Simplifying and Isolating Failure-Inducing Input
IEEE Transactions on Software Engineering
Constructing test suites for interaction testing
Proceedings of the 25th International Conference on Software Engineering
Bug isolation via remote program sampling
PLDI '03 Proceedings of the ACM SIGPLAN 2003 conference on Programming language design and implementation
An Investigation of the Applicability of Design of Experiments to Software Testing
SEW '02 Proceedings of the 27th Annual NASA Goddard Software Engineering Workshop (SEW-27'02)
Variable Strength Interaction Testing of Components
COMPSAC '03 Proceedings of the 27th Annual International Conference on Computer Software and Applications
Skoll: Distributed Continuous Quality Assurance
Proceedings of the 26th International Conference on Software Engineering
Covering arrays for efficient fault characterization in complex configuration spaces
ISSTA '04 Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis
Proceedings of the 27th international conference on Software engineering
The design of the TAO real-time object request broker
Computer Communications
Coverage and adequacy in software product line testing
Proceedings of the ISSTA 2006 workshop on Role of software architecture for testing and analysis
Testing across configurations: implications for combinatorial testing
ACM SIGSOFT Software Engineering Notes
IEEE Transactions on Software Engineering
Interaction testing of highly-configurable systems in the presence of constraints
Proceedings of the 2007 international symposium on Software testing and analysis
One-test-at-a-time heuristic search for interaction test suites
Proceedings of the 9th annual conference on Genetic and evolutionary computation
Towards incremental adaptive covering arrays
Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering
Test suite prioritization by interaction coverage
Workshop on Domain specific approaches to software test automation: in conjunction with the 6th ESEC/FSE joint meeting
Towards incremental adaptive covering arrays
The 6th Joint Meeting on European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering: companion papers
Generating combinatorial test suite for interaction relationship
Fourth international workshop on Software quality assurance: in conjunction with the 6th ESEC/FSE joint meeting
Covering array sampling of input event sequences for automated gui testing
Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering
Configuration-aware regression testing: an empirical study of sampling and prioritization
ISSTA '08 Proceedings of the 2008 international symposium on Software testing and analysis
A backtracking search tool for constructing combinatorial test suites
Journal of Systems and Software
Merging covering arrays and compressing multiple sequence alignments
Discrete Applied Mathematics
Incremental covering array failure characterization in large configuration spaces
Proceedings of the eighteenth international symposium on Software testing and analysis
Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing
TAP '09 Proceedings of the 3rd International Conference on Tests and Proofs
Interaction Coverage Meets Path Coverage by SMT Constraint Solving
TESTCOM '09/FATES '09 Proceedings of the 21st IFIP WG 6.1 International Conference on Testing of Software and Communication Systems and 9th International FATES Workshop
A logic-based approach to combinatorial testing with constraints
TAP'08 Proceedings of the 2nd international conference on Tests and proofs
Reducing field failures in system configurable software: cost-based prioritization
ISSRE'09 Proceedings of the 20th IEEE international conference on software reliability engineering
Community-based, collaborative testing and analysis
Proceedings of the FSE/SDP workshop on Future of software engineering research
A survey of combinatorial testing
ACM Computing Surveys (CSUR)
Verification of general and cyclic covering arrays using grid computing
Globe'10 Proceedings of the Third international conference on Data management in grid and peer-to-peer systems
A Formal Logic Approach to Constrained Combinatorial Testing
Journal of Automated Reasoning
Advances in Software Engineering - Special issue on software test automation
Evaluating improvements to a meta-heuristic search for constrained interaction testing
Empirical Software Engineering
The Minimal Failure-Causing Schema of Combinatorial Testing
ACM Transactions on Software Engineering and Methodology (TOSEM)
Feedback driven adaptive combinatorial testing
Proceedings of the 2011 International Symposium on Software Testing and Analysis
Characterizing failure-causing parameter interactions by adaptive testing
Proceedings of the 2011 International Symposium on Software Testing and Analysis
Using feature locality: can we leverage history to avoid failures during reconfiguration?
Proceedings of the 8th workshop on Assurances for self-adaptive systems
A variable strength interaction test suites generation strategy using Particle Swarm Optimization
Journal of Systems and Software
Combining model-based and combinatorial testing for effective test case generation
Proceedings of the 2012 International Symposium on Software Testing and Analysis
iTree: efficiently discovering high-coverage configurations using interaction trees
Proceedings of the 34th International Conference on Software Engineering
Faulty interaction identification via constraint solving and optimization
SAT'12 Proceedings of the 15th international conference on Theory and Applications of Satisfiability Testing
Supercomputing and grid computing on the verification of covering arrays
The Journal of Supercomputing
T-wise combinatorial interaction test suites construction based on coverage inheritance
Software Testing, Verification & Reliability
Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering
An orchestrated survey of methodologies for automated software test case generation
Journal of Systems and Software
Exploring feature interactions in the wild: the new feature-interaction challenge
Proceedings of the 5th International Workshop on Feature-Oriented Software Development
Hi-index | 0.01 |
Many modern software systems are designed to be highly configurable so they can run on and be optimized for a wide variety of platforms and usage scenarios. Testing such systems is difficult because, in effect, you are testing a multitude of systems, not just one. Moreover, bugs can and do appear in some configurations, but not in others. Our research focuses on a subset of these bugs that are "option-related”—those that manifest with high probability only when specific configuration options take on specific settings. Our goal is not only to detect these bugs, but also to automatically characterize the configuration subspaces (i.e., the options and their settings) in which they manifest. To improve efficiency, our process tests only a sample of the configuration space, which we obtain from mathematical objects called covering arrays. This paper compares two different kinds of covering arrays for this purpose and assesses the effect of sampling strategy on fault characterization accuracy. Our results strongly suggest that sampling via covering arrays allows us to characterize option-related failures nearly as well as if we had tested exhaustively, but at a much lower cost. We also provide guidelines for using our approach in practice.