Automatic Generation of Test Oracles—From Pilot Studies to Application

  • Authors:

  • Martin S. Feather;Ben Smith

  • Affiliations:

  • Jet Propulsion Laboratory, California Institute of Technology, 4800 Oak Grove Drive, Pasadena, CA 91109, USA. martin.s.feather@jpl.nasa.gov;Jet Propulsion Laboratory, California Institute of Technology, 4800 Oak Grove Drive, Pasadena, CA 91109, USA. ben.d.smith@jpl.nasa.gov

  • Venue:
  • Automated Software Engineering
  • Year:
  • 2001

Quantified Score

Hi-index 0.00

Visualization

Abstract

We describe a progression from pilot studies to development and use of domain-specific verification and validation (V&V) automation. Our domain is the testing of an AI planning system that forms a key component of an autonomous spacecraft. We used pilot studies to ascertain opportunities for, and suitability of, automating various analyses whose results would contribute to V&V in our domain. These studies culminated in development of an automatic generator of automated test oracles. This was then applied and extended in the course of testing the spacecraft's AI planning system.Richardson et al. (1992, In Proceedings of the 14th International Conference on Software Engineering, Melbourne, Australia, pp. 105–118), presents motivation for automatic test oracles, and considered the issues and approaches particular to test oracles derived from specifications. Our work, carried through from conception to application, confirms many of their insights. Generalizing from our specific domain, we present some additional insights and recommendations concerning the use of test oracles for V&V of knowledge-based systems.