Execution generated test cases: how to make systems code crash itself

Authors:
Cristian Cadar;Dawson Engler
Affiliations:
Computer Systems Laboratory, Stanford University, Stanford, CA;Computer Systems Laboratory, Stanford University, Stanford, CA
Venue:
SPIN'05 Proceedings of the 12th international conference on Model Checking Software
Year:
2005

Citing 19
Cited 50

An empirical study of the reliability of UNIX utilities

Communications of the ACM
The chaining approach for software test data generation

ACM Transactions on Software Engineering and Methodology (TOSEM)
The Model Checker SPIN

IEEE Transactions on Software Engineering - Special issue on formal methods in software practice
Model checking for programming languages using VeriSoft

Proceedings of the 24th ACM SIGPLAN-SIGACT symposium on Principles of programming languages
Automatic test data generation using constraint solving techniques

Proceedings of the 1998 ACM SIGSOFT international symposium on Software testing and analysis
Automated test data generation using an iterative relaxation method

SIGSOFT '98/FSE-6 Proceedings of the 6th ACM SIGSOFT international symposium on Foundations of software engineering
Bandera: extracting finite-state models from Java source code

Proceedings of the 22nd international conference on Software engineering
A static analyzer for finding dynamic programming errors

Software—Practice & Experience
Automatic predicate abstraction of C programs

Proceedings of the ACM SIGPLAN 2001 conference on Programming language design and implementation
Automatically validating temporal safety properties of interfaces

SPIN '01 Proceedings of the 8th international SPIN workshop on Model checking of software
Analysis of the constraint solver in UNA based test data generation

Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineering
ESP: path-sensitive program verification in polynomial time

PLDI '02 Proceedings of the ACM SIGPLAN 2002 Conference on Programming language design and implementation
Korat: automated testing based on Java predicates

ISSTA '02 Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis
CIL: Intermediate Language and Tools for Analysis and Transformation of C Programs

CC '02 Proceedings of the 11th International Conference on Compiler Construction
From Code to Models

ACSD '01 Proceedings of the Second International Conference on Application of Concurrency to System Design
Model Checking Programs

ASE '00 Proceedings of the 15th IEEE international conference on Automated software engineering
SELECT—a formal system for testing and debugging programs by symbolic execution

Proceedings of the international conference on Reliable software
DART: directed automated random testing

Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation
Enhancing server availability and security through failure-oblivious computing

OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6

CUTE: a concolic unit testing engine for C

Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering
On deriving unknown vulnerabilities from zero-day polymorphic and metamorphic worm exploits

Proceedings of the 12th ACM conference on Computer and communications security
Software partitioning for effective automated unit testing

EMSOFT '06 Proceedings of the 6th ACM & IEEE International conference on Embedded software
Compositional dynamic test generation

Proceedings of the 34th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages
PathExpander: Architectural Support for Increasing the Path Coverage of Dynamic Bug Detection

Proceedings of the 39th Annual IEEE/ACM International Symposium on Microarchitecture
Hybrid Concolic Testing

ICSE '07 Proceedings of the 29th international conference on Software Engineering
Under-constrained execution: making automatic code destruction easy and scalable

Proceedings of the 2007 international symposium on Software testing and analysis
Variably interprocedural program analysis for runtime error detection

Proceedings of the 2007 international symposium on Software testing and analysis
Dynamic test input generation for database applications

Proceedings of the 2007 international symposium on Software testing and analysis
State space exploration using feedback constraint generation and Monte-Carlo sampling

Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering
Assertion-based repair of complex data structures

Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering
Mining library specifications using inductive logic programming

Proceedings of the 30th international conference on Software engineering
Handling dynamic data structures in search based testing

Proceedings of the 10th annual conference on Genetic and evolutionary computation
Dynamic test input generation for web applications

ISSTA '08 Proceedings of the 2008 international symposium on Software testing and analysis
Finding bugs in dynamic web applications

ISSTA '08 Proceedings of the 2008 international symposium on Software testing and analysis
Dynamic inference of likely data preconditions over predicates by tree learning

ISSTA '08 Proceedings of the 2008 international symposium on Software testing and analysis
EXE: Automatically Generating Inputs of Death

ACM Transactions on Information and System Security (TISSEC)
Whispec: white-box testing of libraries using declarative specifications

LCSD '07 Proceedings of the 2007 Symposium on Library-Centric Software Design
Test Input Generation for Programs with Pointers

TACAS '09 Proceedings of the 15th International Conference on Tools and Algorithms for the Construction and Analysis of Systems: Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2009,
Loop-extended symbolic execution on binary programs

Proceedings of the eighteenth international symposium on Software testing and analysis
RWset: attacking path explosion in constraint-based test generation

TACAS'08/ETAPS'08 Proceedings of the Theory and practice of software, 14th international conference on Tools and algorithms for the construction and analysis of systems
AFID: an automated approach to collecting software faults

Automated Software Engineering
KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs

OSDI'08 Proceedings of the 8th USENIX conference on Operating systems design and implementation
Input generation via decomposition and re-stitching: finding bugs in Malware

Proceedings of the 17th ACM conference on Computer and communications security
An empirical investigation into branch coverage for C programs using CUTE and AUSTIN

Journal of Systems and Software
Symbolic execution for software testing in practice: preliminary assessment

Proceedings of the 33rd International Conference on Software Engineering
Directed incremental symbolic execution

Proceedings of the 32nd ACM SIGPLAN conference on Programming language design and implementation
FlagRemover: A testability transformation for transforming loop-assigned flags

ACM Transactions on Software Engineering and Methodology (TOSEM)
A Parallel Approach to Concolic Testing with Low-cost Synchronization

Electronic Notes in Theoretical Computer Science (ENTCS)
New ideas track: testing mapreduce-style programs

Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering
MACE: model-inference-assisted concolic exploration for protocol and vulnerability discovery

SEC'11 Proceedings of the 20th USENIX conference on Security
Practical, low-effort equivalence verification of real code

CAV'11 Proceedings of the 23rd international conference on Computer aided verification
Linear obfuscation to combat symbolic execution

ESORICS'11 Proceedings of the 16th European conference on Research in computer security
Symbolic execution of alloy models

ICFEM'11 Proceedings of the 13th international conference on Formal methods and software engineering
Path-exploration lifting: hi-fi tests for lo-fi emulators

ASPLOS XVII Proceedings of the seventeenth international conference on Architectural Support for Programming Languages and Operating Systems
Scalable testing of file system checkers

Proceedings of the 7th ACM european conference on Computer Systems
Software model checking: searching for computations in the abstract or the concrete

IFM'05 Proceedings of the 5th international conference on Integrated Formal Methods
Symbolic search-based testing

ASE '11 Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering
Staged symbolic execution

Proceedings of the 27th Annual ACM Symposium on Applied Computing
Residual investigation: predictive and precise bug detection

Proceedings of the 2012 International Symposium on Software Testing and Analysis
STING: finding name resolution vulnerabilities in programs

Security'12 Proceedings of the 21st USENIX conference on Security symposium
AUSTIN: An open source tool for search based software testing of C programs

Information and Software Technology
Scaling symbolic execution using ranged analysis

Proceedings of the ACM international conference on Object oriented programming systems languages and applications
Test input generation using dynamic programming

Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering
Symbolic execution for software testing: three decades later

Communications of the ACM
Sigma*: symbolic learning of input-output specifications

POPL '13 Proceedings of the 40th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages
State of the art: Dynamic symbolic execution for automated test generation

Future Generation Computer Systems
KATCH: high-coverage testing of software patches

Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering
Scaling symbolic execution using staged analysis

Innovations in Systems and Software Engineering
Towards a lazier symbolic pathfinder

ACM SIGSOFT Software Engineering Notes

Quantified Score

Hi-index	0.02

Visualization

Abstract

This paper presents a technique that uses code to automatically generate its own test cases at run time by using a combination of symbolic and concrete (i.e regular) execution The input values to a program (or software component) provide the standard interface of any testing framework with the program it is testing and generating input values that will explore all the “interesting” behavior in the tested program remains an important open problem in software testing research. Our approach works by turning the problem on its head: we lazily generate from within the program itself the input values to the program (and values derived from input values) as needed. We applied the technique to real code and found numerous corner case errors ranging from simple memory overflows and infinite loops to subtle issues in the interpretation of language standards.