Accurate estimates without calibration?

  • Authors:
  • Tim Menzies;Oussama Elrawas;Barry Boehm;Raymond Madachy;Jairus Hihn;Daniel Baker;Karen Lum

  • Affiliations:
  • LCSEE, West Virginia University, Morgantown, WV;LCSEE, West Virginia University, Morgantown, WV;CS, University of Southern California, Los Angeles, California;CS, University of Southern California, Los Angeles, California;JPL, California;LCSEE, West Virginia University, Morgantown, WV;JPL, California

  • Venue:
  • ICSP'08 Proceedings of the Software process, 2008 international conference on Making globally distributed software development a success story
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

Most process models calibrate their internal settings using historicaldata. Collecting this data is expensive, tedious, and often an incomplete process. Is it possible to make accurate software process estimates without historicaldata? Suppose much of uncertainty in a model comes from a small subset of themodel variables. If so, then after (a) ranking variables by their ability to constrainthe output; and (b) applying a small number of the top-ranked variables; then itshould be possible to (c) make stable predictions in the constrained space. To test that hypothesis, we combined a simulated annealer (to generate randomsolutions) with a variable ranker. The results where quite dramatic: in one ofthe studies in this paper, we found process options that reduced the median andvariance of the effort estimates by a factor of 20. In ten case studies, we show thatthe estimates generated in this manner are usually similar to those produced bystandard local calibration. Our conclusion is that while it is always preferable to tune models to localdata, it is possible to learn process control options without that data.