Evolutionary functional black-box testing in an industrial setting

Authors:
Tanja E. Vos;Felix F. Lindlar;Benjamin Wilmes;Andreas Windisch;Arthur I. Baars;Peter M. Kruse;Hamilton Gross;Joachim Wegener
Affiliations:
Research Center on Software Production Methods, Technical University of Valencia, Valencia, Spain;Daimler Center for Automotive IT Innovations, Berlin Institute of Technology, Berlin, Germany;Daimler Center for Automotive IT Innovations, Berlin Institute of Technology, Berlin, Germany;Daimler Center for Automotive IT Innovations, Berlin Institute of Technology, Berlin, Germany;Research Center on Software Production Methods, Technical University of Valencia, Valencia, Spain;Berner & Mattner Systemtechnik GmbH, Berlin, Germany;Berner & Mattner Systemtechnik GmbH, Berlin, Germany;Berner & Mattner Systemtechnik GmbH, Berlin, Germany
Venue:
Software Quality Control
Year:
2013

Citing 27
Cited 0

Software testing techniques (2nd ed.)

Software testing techniques (2nd ed.)
Software test automation: effective use of test execution tools

Software test automation: effective use of test execution tools
Empirical studies of software engineering: a roadmap

Proceedings of the Conference on The Future of Software Engineering
Automated test-data generation for exception conditions

Software—Practice & Experience
Genetic Algorithms in Search, Optimization and Machine Learning

Genetic Algorithms in Search, Optimization and Machine Learning
Preliminary guidelines for empirical research in software engineering

IEEE Transactions on Software Engineering
Automatic Test Data Generation For Structural Testing Of Embedded Software Systems By Evolutionary Testing

GECCO '02 Proceedings of the Genetic and Evolutionary Computation Conference
Improving Evolutionary Testing By Flag Removal

GECCO '02 Proceedings of the Genetic and Evolutionary Computation Conference
Evolving Objects: A General Purpose Evolutionary Computation Library

Selected Papers from the 5th European Conference on Artificial Evolution
A Comparison of Static Analysis and Evolutionary Testing for the Verification of Timing Constraints

RTAS '98 Proceedings of the Fourth IEEE Real-Time Technology and Applications Symposium
Reviewing 25 Years of Testing Technique Experiments

Empirical Software Engineering
Case Studies for Software Engineers

SEW '05 Proceedings of the 29th Annual IEEE/NASA Software Engineering Workshop - Tutorial Notes
Studying Software Engineers: Data Collection Techniques for Software Field Studies

Empirical Software Engineering
Search-based software test data generation: a survey: Research Articles

Software Testing, Verification & Reliability
Improving evolutionary real-time testing

Proceedings of the 8th annual conference on Genetic and evolutionary computation
A Critical Analysis of Empirical Research in Software Testing

ESEM '07 Proceedings of the First International Symposium on Empirical Software Engineering and Measurement
Evolutionary functional testing

Computers and Operations Research
Adaptive operator selection with dynamic multi-armed bandits

Proceedings of the 10th annual conference on Genetic and evolutionary computation
Multi-objective Improvement of Software Using Co-evolution and Smart Seeding

SEAL '08 Proceedings of the 7th International Conference on Simulated Evolution and Learning
Evolutionary White-Box Software Test with the EvoTest Framework: A Progress Report

ICSTW '09 Proceedings of the IEEE International Conference on Software Testing, Verification, and Validation Workshops
Signal Generation for Search-Based Testing of Continuous Systems

ICSTW '09 Proceedings of the IEEE International Conference on Software Testing, Verification, and Validation Workshops
A highly configurable test system for evolutionary black-box testing of embedded systems

Proceedings of the 11th Annual conference on Genetic and evolutionary computation
Evolutionary functional testing of continuous control systems

Proceedings of the 11th Annual conference on Genetic and evolutionary computation
Structural and functional sequence test of dynamic and state-based software with evolutionary algorithms

GECCO'03 Proceedings of the 2003 international conference on Genetic and evolutionary computation: PartII
Integrating Model-Based Testing with Evolutionary Functional Testing

ICSTW '10 Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops
Industrial Scaled Automated Structural Testing with the Evolutionary Testing Tool

ICST '10 Proceedings of the 2010 Third International Conference on Software Testing, Verification and Validation
Search-Based Software Testing: Past, Present and Future

ICSTW '11 Proceedings of the 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops

Quantified Score

Hi-index	0.00

Visualization

Abstract

During the past years, evolutionary testing research has reported encouraging results for automated functional (i.e. black-box) testing. However, despite promising results, these techniques have hardly been applied to complex, real-world systems and as such, little is known about their scalability, applicability, and acceptability in industry. In this paper, we describe the empirical setup used to study the use of evolutionary functional testing in industry through two case studies, drawn from serial production development environments at Daimler and Berner & Mattner Systemtechnik, respectively. Results of the case studies are presented, and research questions are assessed based on them. In summary, the results indicate that evolutionary functional testing in an industrial setting is both scalable and applicable. However, the creation of fitness functions is time-consuming. Although in some cases, this is compensated by the results, it is still a significant factor preventing functional evolutionary testing from more widespread use in industry.