Reducing wasted development time via continuous testing

  • Authors:

  • David Saff;Michael D. Ernst

  • Affiliations:

  • -;-

  • Venue:
  • ISSRE '03 Proceedings of the 14th International Symposium on Software Reliability Engineering
  • Year:
  • 2003

Quantified Score

Hi-index 0.00

Visualization

Abstract

Testing is often performed frequently during developmentto ensure software reliability by catching regressionerrors quickly. However, stopping frequently to test alsowastes time by holding up development progress. User studieson real development projects indicate that these twosources of wasted time account for 10-15% of developmenttime. These measurements use a novel technique for computingthe wasted extra development time incurred by a delayin discovering a regression error.We present a model of developer behavior that infersdeveloper beliefs from developer behavior, and that predictsdeveloper behavior in new environments-in particular,when changing testing methodologies or tools to reducewasted time. Changing test ordering or reporting reduceswasted time by 4-41% in our case study. Changingthe frequency with which tests are run can reduce wastedtime by 31-82% (but developers cannot know the ideal frequencyexcept after the fact). We introduce and evaluate anew technique, continuous testing, that uses spare CPU resourcesto continuously run tests in the background, providingrapid feedback about test failures as as source code isedited. Continuous testing reduced wasted time by 92-98%,a substantial improvement over the other approaches.We have integrated continuous testing into two developmentenvironments, and are beginning user studies to evaluateits efficacy. We believe it has the potential to reduce thecost and improve the efficacy of testing and, as a result, toimprove the reliability of delivered systems.