Software Engineering Journal - Controlling software projects
Art of Software Testing
Formal Test Automation: A Simple Experiment
Proceedings of the IFIP TC6 12th International Workshop on Testing Communicating Systems: Method and Applications
The AGEDIS tools for model based testing
ISSTA '04 Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis
A Model Checking Language for Concurrent Value-Passing Systems
FM '08 Proceedings of the 15th international symposium on Formal Methods
EMSOFT '08 Proceedings of the 8th ACM international conference on Embedded software
Timed Testing under Partial Observability
ICST '09 Proceedings of the 2009 International Conference on Software Testing Verification and Validation
Interaction Coverage Meets Path Coverage by SMT Constraint Solving
TESTCOM '09/FATES '09 Proceedings of the 21st IFIP WG 6.1 International Conference on Testing of Software and Communication Systems and 9th International FATES Workshop
CADP 2006: a toolbox for the construction and analysis of distributed processes
CAV'07 Proceedings of the 19th international conference on Computer aided verification
Model based testing with labelled transition systems
Formal methods and testing
Model-based testing of object-oriented reactive systems with spec explorer
Formal methods and testing
FMICS'10 Proceedings of the 15th international conference on Formal methods for industrial critical systems
FMICS'11 Proceedings of the 16th international conference on Formal methods for industrial critical systems
Online testing of real-time systems using UPPAAL
FATES'04 Proceedings of the 4th international conference on Formal Approaches to Software Testing
A symbolic framework for model-based testing
FATES'06/RV'06 Proceedings of the First combined international conference on Formal Approaches to Software Testing and Runtime Verification
JTorX: a tool for on-line model-driven test derivation and execution
TACAS'10 Proceedings of the 16th international conference on Tools and Algorithms for the Construction and Analysis of Systems
Hi-index | 0.00 |
We report on the actual industrial use of formal methods during the development of a software bus. During an internship at Neopost Inc., of 14 weeks, we developed the server component of a software bus, called the XBus, using formal methods during the design, validation and testing phase: we modeled our design of the XBus in the process algebra mCRL2, validated the design using the mCRL2-simulator, and fully automatically tested our implementation with the model-based test tool JTorX. This resulted in a well-tested software bus with a maintainable architecture. Writing the model (m"d"e"v), simulating it, and testing the implementation with JTorX only took 17% of the total development time. Moreover, the errors found with model-based testing would have been hard to find with conventional test methods. Thus, we show that formal engineering can be feasible, beneficial and cost-effective. The findings above, reported earlier by us in (Sijtema et al., 2011) [1], were well-received, also in industrially oriented conferences (Ferreira and Romanenko, 2010) [2] and [3]. In this paper, we look back on the case study, and carefully analyze its merits and shortcomings. We reflect on (1) the added benefits of model checking, (2) model completeness and (3) the quality and performance of the test process. Thus, in a second phase, after the internship, we model checked the XBus protocol-this was not done in [1] since the Neopost business process required a working implementation after 14 weeks. We used the CADP tool evaluator4 to check the behavioral requirements obtained during the development. Model checking did not uncover errors in model m"d"e"v, but revealed that model m"d"e"v was neither complete nor optimized: in particular, requirements to the so-called bad weather behavior (exceptions, unexpected inputs, etc.) were missing. Therefore, we created several improved models, checked that we could validate them, and used them to analyze quality and performance of the test process. Model checking was expensive: it took us approx. 4 weeks in total, compared to 3 weeks for the entire model-based testing approach during the internship. In the second phase, we analyzed the quality and performance of the test process, where we looked at both code and model coverage. We found that high code coverage (almost 100%) is in most cases obtained within 1000 test steps and 2 minutes, which matches the fact that the faults in the XBus were discovered within a few minutes. Summarizing, we firmly believe that the formal engineering approach is cost-effective, and produces high quality software products. Model checking does yield significantly better models, but is also costly. Thus, system developers should trade off higher model quality against higher costs.