Reducing qualitative human oracle costs associated with automatically generated test data

  • Authors:

  • Phil McMinn;Mark Stevenson;Mark Harman

  • Affiliations:

  • The University of Sheffield, Regent Court, Portobello, Sheffield, UK;The University of Sheffield, Regent Court, Portobello, Sheffield, UK;CREST, King's College London, Strand, London, UK

  • Venue:
  • Proceedings of the First International Workshop on Software Test Output Validation
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

Due to the frequent non-existence of an automated oracle, test cases are often evaluated manually in practice. However, this fact is rarely taken into account by automatic test data generators, which seek to maximise a program's structural coverage only. The test data produced tends to be of a poor fit with the program's operational profile. As a result, each test case takes longer for a human to check, because the scenarios that arbitrary-looking data represent require time and effort to understand. This short paper proposes methods to extracting knowledge from programmers, source code and documentation and its incorporation into the automatic test data generation process so as to inject the realism required to produce test cases that are quick and easy for a human to comprehend and check. The aim is to reduce the so-called qualitative human oracle costs associated with automatic test data generation. The potential benefits of such an approach are demonstrated with a simple case study.