Using Empirical Testbeds to Accelerate Technology Maturity and Transition: The SCRover Experience

  • Authors:
  • Barry Boehm;Jesal Bhuta;David Garlan;Eric Gradman;LiGuo Huang;Alexander Lam;Ray Madachy;Nenad Medvidovic;Kenneth Meyer;Steven Meyers;Gustavo Perez;Kirk Reinholtz;Roshanak Roshandel;Nicolas Rouquette

  • Affiliations:
  • University of Southern California;University of Southern California;Carnegie Mellon University;University of Southern California;University of Southern California;University of Southern California;University of Southern California;University of Southern California;Jet Propulsion Laboratory;University of Southern California;University of Southern California;Jet Propulsion Laboratory;University of Southern California;Jet Propulsion Laboratory

  • Venue:
  • ISESE '04 Proceedings of the 2004 International Symposium on Empirical Software Engineering
  • Year:
  • 2004

Quantified Score

Hi-index 0.00

Visualization

Abstract

This paper is an experience report on a first attempt to develop and apply a new form of software: a full-service empirical testbed designed to evaluate alternative software dependability technologies, and to accelerate their maturation and transition into project use. The SCRover testbed includes not only the specifications, code, and hardware of a public safety robot, but also the package of instrumentation, scenario drivers, seeded defects, experimentation guidelines, and comparative effort and defect data needed to facilitate technology evaluation experiments. The SCRover testbedýs initial operational capability has been recently applied to empirically evaluate two architecture definition languages (ADLs) and toolsets, Mae and AcmeStudio. The testbed evaluation showed (1) that the ADL-based toolsets were complementary and cost-effective to apply to mission-critical systems; (2) that the testbed was cost-effective to use by researchers; and (3) that collaboration in testbed use by researchers and the Jet Propulsion Laboratory (JPL) project users resulted in actions to accelerate technology maturity and transition into project use. The evaluation also identified a number of lessons learned for improving the SCRover testbed, and for development and application of future technology evaluation testbeds.