The search for the laws of automatic random testing

Authors:
Carlo A. Furia;Bertrand Meyer;Manuel Oriol;Andrey Tikhomirov;Yi Wei
Affiliations:
ETH Zurich, Zurich, Switzerland;ETH Zurich, Zurich, Switzerland;ABB Corporate Research, Industrial Software Systems, Baden-Dättwil, Switzerland and University of York, York, UK;Software Engineering Laboratory, ITMO, st. Petersburg, Russia;ETH Zurich, Zurich, Switzerland
Venue:
Proceedings of the 28th Annual ACM Symposium on Applied Computing
Year:
2013

Citing 17
Cited 0

QuickCheck: a lightweight tool for random testing of Haskell programs

ICFP '00 Proceedings of the fifth ACM SIGPLAN international conference on Functional programming
Software Engineering Economics

Software Engineering Economics
Art of Software Testing

Art of Software Testing
JCrasher: an automatic robustness tester for Java

Software—Practice & Experience
Tool support for randomized unit testing

Proceedings of the 1st international workshop on Random testing
Some upper and lower bounds on the coupon collector problem

Journal of Computational and Applied Mathematics
Feedback-Directed Random Test Generation

ICSE '07 Proceedings of the 29th international conference on Software Engineering
Programs That Test Themselves

Computer
A model of large program development

IBM Systems Journal
Formal analysis of the effectiveness and predictability of random testing

Proceedings of the 19th international symposium on Software testing and analysis
Testing .NET Code with YETI

ICECCS '10 Proceedings of the 2010 15th IEEE International Conference on Engineering of Complex Computer Systems
On the number and nature of faults found by random testing

Software Testing, Verification & Reliability
A practical guide for using statistical tests to assess randomized algorithms in software engineering

Proceedings of the 33rd International Conference on Software Engineering
Programs, tests, and oracles: the foundations of testing revisited

Proceedings of the 33rd International Conference on Software Engineering
Testing container classes: random or systematic?

FASE'11/ETAPS'11 Proceedings of the 14th international conference on Fundamental approaches to software engineering: part of the joint European conferences on theory and practice of software
Eclat: automatic generation and classification of test inputs

ECOOP'05 Proceedings of the 19th European conference on Object-Oriented Programming
Random Testing: Evaluation of a Law Describing the Number of Faults Found

ICST '12 Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation

Quantified Score

Hi-index	0.00

Visualization

Abstract

Can one estimate the number of remaining faults in a software system? A credible estimation technique would be immensely useful to project managers as well as customers. It would also be of theoretical interest, as a general law of software engineering. We investigate possible answers in the context of automated random testing, a method that is increasingly accepted as an effective way to discover faults. Our experimental results, derived from best-fit analysis of a variety of mathematical functions, based on a large number of automated tests of library code equipped with automated oracles in the form of contracts, suggest a poly-logarithmic law. Although further confirmation remains necessary on different code bases and testing techniques, we argue that understanding the laws of testing may bring significant benefits for estimating the number of detectable faults and comparing different projects and practices.