An experimental evaluation of the assumption of independence in multiversion programming
IEEE Transactions on Software Engineering
Run-time consistency checking of algebraic specifications
TAV4 Proceedings of the symposium on Testing, analysis, and verification
C4.5: programs for machine learning
C4.5: programs for machine learning
Designing programs that check their work
Journal of the ACM (JACM)
The nature of statistical learning theory
The nature of statistical learning theory
Fast training of support vector machines using sequential minimal optimization
Advances in kernel methods
Dynamically discovering likely program invariants to support program evolution
Proceedings of the 21st international conference on Software engineering
Alloy: a lightweight object modelling notation
ACM Transactions on Software Engineering and Methodology (TOSEM)
Korat: automated testing based on Java predicates
ISSTA '02 Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis
Semi-proving: an integrated method based on global symbolic evaluation and metamorphic testing
ISSTA '02 Proceedings of the 2002 ACM SIGSOFT international symposium on Software testing and analysis
Tracking down software bugs using automatic anomaly detection
Proceedings of the 24th International Conference on Software Engineering
Pseudo-oracles for non-testable programs
ACM '81 Proceedings of the ACM '81 conference
Software Testing Using Algebraic Specification Based Test Oracles
Software Testing Using Algebraic Specification Based Test Oracles
COMPSAC '03 Proceedings of the 27th Annual International Conference on Computer Software and Applications
The design and implementation of Zap: a system for migrating computing environments
OSDI '02 Proceedings of the 5th symposium on Operating systems design and implementationCopyright restrictions prevent ACM from being able to make the PDFs for this conference available for downloading
Is mutation an appropriate tool for testing experiments?
Proceedings of the 27th international conference on Software engineering
Software Manual for the Elementary Functions (Prentice-Hall series in computational mathematics)
Software Manual for the Elementary Functions (Prentice-Hall series in computational mathematics)
Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems)
Parameterizing random test data according to equivalence classes
Proceedings of the 2nd international workshop on Random testing: co-located with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2007)
QSIC '07 Proceedings of the Seventh International Conference on Quality Software
Quality Assurance of Software Applications Using the In Vivo Testing Approach
ICST '09 Proceedings of the 2009 International Conference on Software Testing Verification and Validation
ICST '09 Proceedings of the 2009 International Conference on Software Testing Verification and Validation
Predicting electricity distribution feeder failures using machine learning susceptibility analysis
IAAI'06 Proceedings of the 18th conference on Innovative applications of artificial intelligence - Volume 2
Checking the conformance of java classes against algebraic specifications
ICFEM'06 Proceedings of the 8th international conference on Formal Methods and Software Engineering
Representation dependence testing using program inversion
Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering
Automated metamorphic testing on the analyses of feature models
Information and Software Technology
Testing a binary space partitioning algorithm with metamorphic testing
Proceedings of the 2011 ACM Symposium on Applied Computing
Metamorphic testing of a Monte Carlo modeling program
Proceedings of the 6th International Workshop on Automation of Software Test
On effective testing of health care simulation software
Proceedings of the 3rd Workshop on Software Engineering in Health Care
The use of mutation in testing experiments and its sensitivity to external threats
Proceedings of the 2011 International Symposium on Software Testing and Analysis
Automating image segmentation verification and validation by learning test oracles
Information and Software Technology
Hi-index | 0.00 |
Metamorphic testing has been shown to be a simple yet effective technique in addressing the quality assurance of applications that do not have test oracles, i.e., for which it is difficult or impossible to know what the correct output should be for arbitrary input. In metamorphic testing, existing test case input is modified to produce new test cases in such a manner that, when given the new input, the application should produce an output that can easily be computed based on the original output. That is, if input x produces output f(x), then we create input x' such that we can predict f(x') based on f(x); if the application does not produce the expected output, then a defect must exist, and either f(x), or f(x') (or both) is wrong. In practice, however, metamorphic testing can be a manually intensive technique for all but the simplest cases. The transformation of input data can be laborious for large data sets, or practically impossible for input that is not in human-readable format. Similarly, comparing the outputs can be error-prone for large result sets, especially when slight variations in the results are not actually indicative of errors (i.e., are false positives), for instance when there is non-determinism in the application and multiple outputs can be considered correct. In this paper, we present an approach called Automated Metamorphic System Testing. This involves the automation of metamorphic testing at the system level by checking that the metamorphic properties of the entire application hold after its execution. The tester is able to easily set up and conduct metamorphic tests with little manual intervention, and testing can continue in the field with minimal impact on the user. Additionally, we present an approach called Heuristic Metamorphic Testing which seeks to reduce false positives and address some cases of non-determinism. We also describe an implementation framework called Amsterdam, and present the results of empirical studies in which we demonstrate the effectiveness of the technique on real-world programs without test oracles.