Reducing field failures in system configurable software: cost-based prioritization

  • Authors:

  • Hema Srikanth;Myra B. Cohen;Xiao Qu

  • Affiliations:

  • IBM Software Group, Westford, MA;Dept. of Comp. Sci & Eng., University of Nebraska-Lincoln, Lincoln, NE;Dept. of Comp. Sci & Eng., University of Nebraska-Lincoln, Lincoln, NE

  • Venue:
  • ISSRE'09 Proceedings of the 20th IEEE international conference on software reliability engineering
  • Year:
  • 2009

Quantified Score

Hi-index 0.00

Visualization

Abstract

System testing of configurable software is an expensive and resource constrained process. Insufficient testing often leads to escaped faults in the field where failures impact customers and are costly to repair. Prior work has shown that it is possible to efficiently sample configurations for testing using combinatorial interaction testing, and to prioritize these configurations to increase the rate of early fault detection. The underlying assumption to date has been that there is no added complexity to configuring a system level environment over a user configurable one; i.e. the time required to setup and test each individual configuration is nominal. In this paper we examine prioritization of system configurable software driven not only by fault detection but also by the cost of configuration and setup time that moving between different configurations incurs. We present a case study on two releases of an enterprise software system using failures reported in the field. We examine the most effective prioritization technique and conclude that (1) using failure history of configurations can improve the early fault detection rate, but that (2) we must consider fault detection rate over time, not by the number of configurations tested. It is better to test related configurations which incur minimal setup time than to test fewer, more diverse configurations.