Using model checking to generate tests from requirements specifications
ESEC/FSE-7 Proceedings of the 7th European software engineering conference held jointly with the 7th ACM SIGSOFT international symposium on Foundations of software engineering
Quickly detecting relevant program invariants
Proceedings of the 22nd international conference on Software engineering
Dynamically Discovering Likely Program Invariants to Support Program Evolution
IEEE Transactions on Software Engineering - Special issue on 1999 international conference on software engineering
Automatic generation of program specifications
ISSTA '02 Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis
Automated Support for Program Refactoring using Invariants
ICSM '01 Proceedings of the IEEE International Conference on Software Maintenance (ICSM'01)
Efficient incremental algorithms for dynamic detection of likely invariants
Proceedings of the 12th ACM SIGSOFT twelfth international symposium on Foundations of software engineering
DART: directed automated random testing
Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation
An overview of JML tools and applications
International Journal on Software Tools for Technology Transfer (STTT) - Special section on formal methods for industrial critical systems
CUTE: a concolic unit testing engine for C
Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering
From daikon to agitator: lessons and challenges in building a commercial tool for developer testing
Proceedings of the 2006 international symposium on Software testing and analysis
Tool-assisted unit-test generation and selection based on operational abstractions
Automated Software Engineering
Software Testing Research: Achievements, Challenges, Dreams
FOSE '07 2007 Future of Software Engineering
Nonparametric Statistics with Applications to Science and Engineering (Wiley Series in Probability and Statistics)
Randoop: feedback-directed random testing for Java
Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion
The Daikon system for dynamic detection of likely invariants
Science of Computer Programming
DySy: dynamic symbolic execution for invariant inference
Proceedings of the 30th international conference on Software engineering
A comparative study of programmer-written and automatically inferred contracts
Proceedings of the eighteenth international symposium on Software testing and analysis
Mutation-driven generation of unit tests and oracles
Proceedings of the 19th international symposium on Software testing and analysis
Proceedings of the 33rd International Conference on Software Engineering
Programs, tests, and oracles: the foundations of testing revisited
Proceedings of the 33rd International Conference on Software Engineering
Discovering likely method specifications
ICFEM'06 Proceedings of the 8th international conference on Formal Methods and Software Engineering
ESC/Java2: uniting ESC/Java and JML
CASSIS'04 Proceedings of the 2004 international conference on Construction and Analysis of Safe, Secure, and Interoperable Smart Devices
Proceedings of the 34th International Conference on Software Engineering
Does automated white-box test generation really help software testers?
Proceedings of the 2013 International Symposium on Software Testing and Analysis
Automated oracles: an empirical study on cost and effectiveness
Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering
Hi-index | 0.00 |
Recently, work has begun on automating the generation of test oracles, which are necessary to fully automate the testing process. One approach to such automation involves dynamic invariant generation which extracts invariants from program executions. To use such invariants as test oracles, however, it is necessary to distinguish correct from incorrect invariants, a process that currently requires human intervention. In this work we examine this process. In particular, we examine the ability of 30 users, across two empirical studies, to classify invariants generated from three Java programs. Our results indicate that users struggle to classify generated invariants: on average, they misclassify 9.1% to 31.7% of correct invariants and 26.1%-58.6% of incorrect invariants. These results contradict prior studies that suggest that classification by users is easy, and indicate that further work needs to be done to bridge the gap between the effectiveness of dynamic invariant generation in theory, and the ability of users to apply it in practice. Along these lines, we suggest several areas for future work.