Applying design of experiments to software testing: experience report
ICSE '97 Proceedings of the 19th international conference on Software engineering
The AETG System: An Approach to Testing Based on Combinatorial Design
IEEE Transactions on Software Engineering
Model-based testing in practice
Proceedings of the 21st international conference on Software engineering
Data mining: practical machine learning tools and techniques with Java implementations
Data mining: practical machine learning tools and techniques with Java implementations
Simplifying and Isolating Failure-Inducing Input
IEEE Transactions on Software Engineering
Proceedings of the 24th International Conference on Software Engineering
A Test Generation Strategy for Pairwise Testing
IEEE Transactions on Software Engineering
Constructing test suites for interaction testing
Proceedings of the 25th International Conference on Software Engineering
Bug isolation via remote program sampling
PLDI '03 Proceedings of the ACM SIGPLAN 2003 conference on Programming language design and implementation
Augmenting Simulated Annealing to Build Interaction Test Suites
ISSRE '03 Proceedings of the 14th International Symposium on Software Reliability Engineering
Software Fault Interactions and Implications for Software Testing
IEEE Transactions on Software Engineering
Skoll: Distributed Continuous Quality Assurance
Proceedings of the 26th International Conference on Software Engineering
Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces
IEEE Transactions on Software Engineering
Interaction testing of highly-configurable systems in the presence of constraints
Proceedings of the 2007 international symposium on Software testing and analysis
Towards incremental adaptive covering arrays
Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering
Community-based, collaborative testing and analysis
Proceedings of the FSE/SDP workshop on Future of software engineering research
Evaluating improvements to a meta-heuristic search for constrained interaction testing
Empirical Software Engineering
Feedback driven adaptive combinatorial testing
Proceedings of the 2011 International Symposium on Software Testing and Analysis
iTree: efficiently discovering high-coverage configurations using interaction trees
Proceedings of the 34th International Conference on Software Engineering
An algorithm for generating t-wise covering arrays from large feature models
Proceedings of the 16th International Software Product Line Conference - Volume 1
Using feature model knowledge to speed up the generation of covering arrays
Proceedings of the Seventh International Workshop on Variability Modelling of Software-intensive Systems
CONFU: Configuration Fuzzing Testing Framework for Software Vulnerability Detection
International Journal of Secure Software Engineering
An orchestrated survey of methodologies for automated software test case generation
Journal of Systems and Software
Hi-index | 0.00 |
The increasing complexity of configurable software systems has created a need for more intelligent sampling mechanisms to detect and characterize failure-inducing dependencies between configurations. Prior work - in idealized environments - has shown that test schedules based on a mathematical object, called a covering array, in combination with classification techniques, can meet this need. Applying this approach in practice, however, is tricky because testing time and resource availability are unpredictable, and because failure characteristics can change from release to release. With current approaches developers must set a key covering array parameter (its strength) based on estimated release times and failure characterizations. This will influence the outcome of their results. In this paper we propose a new approach that incrementally builds covering array schedules. This approach begins at a low strength, and then iteratively increases strength as resources allow. At each stage previously tested configurations are reused, thus avoiding duplication of work. With the incremental approach developers need never commit to a specific covering array strength. Instead, by using progressively stronger covering array schedules, failures due to few configuration dependencies can be found and classified as soon and as cheaply as possibly. Additionally, it eliminates the risks of committing to overly strong test schedules. We evaluate this new approach through a case study on three consecutive releases of MySQL, an open source database. Our results suggest that our approach is as good or better than previous approaches, costing less in most cases, and allowing greater flexibility in environments with unpredictable development constraints.