Qualitative Methods in Empirical Studies of Software Engineering
IEEE Transactions on Software Engineering
Preliminary guidelines for empirical research in software engineering
IEEE Transactions on Software Engineering
Automated generation of testcase datasets
Proceedings of the international conference on Reliable software
Evolutionary testing of classes
ISSTA '04 Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis
JCrasher: an automatic robustness tester for Java
Software—Practice & Experience
Search-based software test data generation: a survey: Research Articles
Software Testing, Verification & Reliability
A Survey of Controlled Experiments in Software Engineering
IEEE Transactions on Software Engineering
Randoop: feedback-directed random testing for Java
Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion
The influence of size and coverage on test suite effectiveness
Proceedings of the eighteenth international symposium on Software testing and analysis
Pex: white box test generation for .NET
TAP'08 Proceedings of the 2nd international conference on Tests and proofs
A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search
IEEE Transactions on Software Engineering
TestFul: An Evolutionary Test Approach for Java
ICST '10 Proceedings of the 2010 Third International Conference on Software Testing, Verification and Validation
Human-competitive results produced by genetic programming
Genetic Programming and Evolvable Machines
Symbolic PathFinder: symbolic execution of Java bytecode
Proceedings of the IEEE/ACM international conference on Automated software engineering
Dsc+Mock: a test case + mock class generator in support of coding against interfaces
Proceedings of the Eighth International Workshop on Dynamic Analysis
An empirical investigation into branch coverage for C programs using CUTE and AUSTIN
Journal of Systems and Software
The Human Competitiveness of Search Based Software Engineering
SSBSE '10 Proceedings of the 2nd International Symposium on Search Based Software Engineering
Exploiting Common Object Usage in Test Case Generation
ICST '11 Proceedings of the 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation
Are automated debugging techniques actually helping programmers?
Proceedings of the 2011 International Symposium on Software Testing and Analysis
EvoSuite: automatic test suite generation for object-oriented software
Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering
MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler
ASE '11 Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering
Mutation-Driven Generation of Unit Tests and Oracles
IEEE Transactions on Software Engineering
Understanding user understanding: determining correctness of generated program invariants
Proceedings of the 2012 International Symposium on Software Testing and Analysis
Sound empirical evidence in software testing
Proceedings of the 34th International Conference on Software Engineering
Proceedings of the 34th International Conference on Software Engineering
Search-based software engineering: Trends, techniques and applications
ACM Computing Surveys (CSUR)
Empirical evaluation of semi-automated XML annotation of text documents with the GoldenGATE editor
ECDL'07 Proceedings of the 11th European conference on Research and Advanced Technology for Digital Libraries
SEAA '12 Proceedings of the 2012 38th Euromicro Conference on Software Engineering and Advanced Applications
IEEE Transactions on Software Engineering
Hi-index | 0.00 |
Automated test generation techniques can efficiently produce test data that systematically cover structural aspects of a program. In the absence of a specification, a common assumption is that these tests relieve a developer of most of the work, as the act of testing is reduced to checking the results of the tests. Although this assumption has persisted for decades, there has been no conclusive evidence to date confirming it. However, the fact that the approach has only seen a limited uptake in industry suggests the contrary, and calls into question its practical usefulness. To investigate this issue, we performed a controlled experiment comparing a total of 49 subjects split between writing tests manually and writing tests with the aid of an automated unit test generation tool, EvoSuite. We found that, on one hand, tool support leads to clear improvements in commonly applied quality metrics such as code coverage (up to 300% increase). However, on the other hand, there was no measurable improvement in the number of bugs actually found by developers. Our results not only cast some doubt on how the research community evaluates test generation tools, but also point to improvements and future work necessary before automated test generation tools will be widely adopted by practitioners.