An empirical comparison between direct and indirect test result checking approaches

Authors:
Peifeng Hu;Zhenyu Zhang;W. K. Chan;T. H. Tse
Affiliations:
The University of Hong Kong, Hong Kong;The University of Hong Kong, Hong Kong;City University of Hong Kong, Hong Kong;The University of Hong Kong, Hong Kong
Venue:
Proceedings of the 3rd international workshop on Software quality assurance
Year:
2006

Citing 30
Cited 2

Software testing techniques (2nd ed.)

Software testing techniques (2nd ed.)
Contracts: specifying behavioral compositions in object-oriented systems

OOPSLA/ECOOP '90 Proceedings of the European conference on object-oriented programming on Object-oriented programming systems, languages, and applications
Checking approximate computations over the reals

STOC '93 Proceedings of the twenty-fifth annual ACM symposium on Theory of computing
Self-testing/correcting with applications to numerical problems

Journal of Computer and System Sciences - Special issue: papers from the 22nd ACM symposium on the theory of computing, May 14–16, 1990
Designing programs that check their work

Journal of the ACM (JACM)
An experimental determination of sufficient mutant operators

ACM Transactions on Software Engineering and Methodology (TOSEM)
Testing object-oriented systems: models, patterns, and tools

Testing object-oriented systems: models, patterns, and tools
Automated test oracles for GUIs

SIGSOFT '00/FSE-8 Proceedings of the 8th ACM SIGSOFT international symposium on Foundations of software engineering: twenty-first century applications
A program testing assistant

Communications of the ACM
Software testing using model programs

Software—Practice & Experience
Semi-proving: an integrated method based on global symbolic evaluation and metamorphic testing

ISSTA '02 Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis
An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing

IEEE Transactions on Software Engineering
Metamorphic Testing of Programs on Partial Differential Equations: A Case Study

COMPSAC '02 Proceedings of the 26th International Computer Software and Applications Conference on Prolonging Software Life: Development and Redevelopment
Automated support for classifying software failure reports

Proceedings of the 25th International Conference on Software Engineering
Automated Metamorphic Testing

COMPSAC '03 Proceedings of the 27th Annual International Conference on Computer Software and Applications
Assertions in programming languages

ACM SIGPLAN Notices
The data mining approach to automated software testing

Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining
Specification-driven automated testing of GUI-based Java programs

ACM-SE 42 Proceedings of the 42nd annual Southeast regional conference
Active learning for automatic classification of software behavior

ISSTA '04 Proceedings of the 2004 ACM SIGSOFT international symposium on Software testing and analysis
Testing Context-Sensitive Middleware-Based Software Applications

COMPSAC '04 Proceedings of the 28th Annual International Computer Software and Applications Conference - Volume 01
Assessing and Improving State-Based Class Testing: A Series of Experiments

IEEE Transactions on Software Engineering
Tree-Based Methods for Classifying Software Failures

ISSRE '04 Proceedings of the 15th International Symposium on Software Reliability Engineering
Is mutation an appropriate tool for testing experiments?

Proceedings of the 27th international conference on Software engineering
MuJava: an automated class mutation system: Research Articles

Software Testing, Verification & Reliability
A Metamorphic Approach to Integration Testing of Context-Sensitive Middleware-Based Applications

QSIC '05 Proceedings of the Fifth International Conference on Quality Software
Towards a Metamorphic Testing Methodology for Service-Oriented Software Applications

QSIC '05 Proceedings of the Fifth International Conference on Quality Software
User guidance for creating precise and accessible property specifications

Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering
Testing context-aware middleware-centric programs: a data flow approach and an RFID-based experimentation

Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering
Automatic goal-oriented classification of failure behaviors for testing XML-based multimedia software applications: An experimental case study

Journal of Systems and Software - Special issue: Quality software
Designing and comparing automated test oracles for GUI-based software applications

ACM Transactions on Software Engineering and Methodology (TOSEM)

An experimental assessment of module documentation-based testing

Information and Software Technology
Metamorphic slice: An application in spectrum-based fault localization

Information and Software Technology

Quantified Score

Hi-index	0.00

Visualization

Abstract

An oracle on software testing is a mechanism for checking whether the system under test has behaved correctly for any executions. In some situations, oracles are unavailable or too expensive to apply. This is known as the oracle problem. It is crucial to develop techniques to address it, and metamorphic testing (MT) was one of such proposals. This paper conducts a controlled experiment to investigate the cost effectiveness of using MT by 38 testers on three open-source programs. The fault detection capability and time cost of MT are compared with the popular assertion checking method. Our results show that MT is cost-efficient and has potentials for detecting more faults than the assertion checking method.