An empirical study of the reliability of UNIX utilities
Communications of the ACM
Partition Testing Does Not Inspire Confidence (Program Testing)
IEEE Transactions on Software Engineering
The chaining approach for software test data generation
ACM Transactions on Software Engineering and Methodology (TOSEM)
On random and partition testing
Proceedings of the 1998 ACM SIGSOFT international symposium on Software testing and analysis
QuickCheck: a lightweight tool for random testing of Haskell programs
ICFP '00 Proceedings of the fifth ACM SIGPLAN international conference on Functional programming
Symbolic execution and program testing
Communications of the ACM
Software Testing Techniques
The Art of Software Testing
JCrasher: an automatic robustness tester for Java
Software—Practice & Experience
DART: directed automated random testing
Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation
Test input generation for java containers using state matching
Proceedings of the 2006 international symposium on Software testing and analysis
EXE: automatically generating inputs of death
Proceedings of the 13th ACM conference on Computer and communications security
Feedback-Directed Random Test Generation
ICSE '07 Proceedings of the 29th international conference on Software Engineering
Randomized Differential Testing as a Prelude to Formal Verification
ICSE '07 Proceedings of the 29th international conference on Software Engineering
An empirical study of the robustness of Windows NT applications using random testing
WSS'00 Proceedings of the 4th conference on USENIX Windows Systems Symposium - Volume 4
Randoop: feedback-directed random testing for Java
Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion
Jartege: a tool for random generation of unit tests for java classes
QoSA'05 Proceedings of the First international conference on Quality of Software Architectures and Software Quality, and Proceedings of the Second International conference on Software Quality
CUTE and jCUTE: concolic unit testing and explicit path model-checking tools
CAV'06 Proceedings of the 18th international conference on Computer Aided Verification
Eclat: automatic generation and classification of test inputs
ECOOP'05 Proceedings of the 19th European conference on Object-Oriented Programming
TestFul: automatic unit-test generation for Java classes
Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 2
MeCC: memory comparison-based clone detector
Proceedings of the 33rd International Conference on Software Engineering
Palus: a hybrid automated test generation tool for java
Proceedings of the 33rd International Conference on Software Engineering
Search-based software testing and test data generation for a dynamic programming language
Proceedings of the 13th annual conference on Genetic and evolutionary computation
ASE '11 Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering
Proceedings of the 34th International Conference on Software Engineering
BugRedux: reproducing field failures for in-house debugging
Proceedings of the 34th International Conference on Software Engineering
Hi-index | 0.00 |
We present a case study in which a team of test engineers at Microsoft applied a feedback-directed random testing tool to a critical component of the .NET architecture. Due to its complexity and high reliability requirements, the component had already been tested by 40 test engineers over five years, using manual testing and many automated testing techniques. Nevertheless, the feedback-directed random testing tool found errors in the component that eluded previous testing, and did so two orders of magnitude faster than a typical test engineer (including time spent inspecting the results of the tool). The tool also led the test team to discover errors in other testing and analysis tools, and deficiencies in previous best-practice guidelines for manual testing. Finally, we identify challenges that random testing faces for continued effectiveness, including an observed decrease in the technique's error detection rate over time.