An Empirical Validation of the Relationship Between the Magnitude of Relative Error and Project Size

• Authors: Erik Stensrud;Tron Foss;Barbara Kitchenham;Ingunn Myrtveit
• Venue: METRICS '02 Proceedings of the 8th International Symposium on Software Metrics
• Year: 2002

Cost estimates are important deliverables of asoftware project. Consequently, a number of costprediction models have been proposed and evaluated.The common evaluation criteria have been MMRE,MdMRE and PRED(k). MRE is the basic metric in theseevaluation criteria. The implicit rationale of using arelative error measure like MRE, rather than an absoluteone, is presumably to have a measure that is independentof project size. In this paper, we investigate if this implicitclaim holds true for several data sets: Albrecht, Kemerer,Finnish, DMR and Accenture-ERP. The results suggestthat MRE is not independent of project size. Rather, MREis larger for small projects than for large projects. Apractical consequence is that a project managerpredicting a small project may falsely believe in a too lowMRE. Vice versa when predicting a large project. Forresearchers, it is important to know that MMRE is not anappropriate measure of the expected MRE of small andlarge projects. We recommend therefore that the data setbe partitioned into two or more subsamples and thatMMRE is reported per subsample. In the long term, weshould consider using other evaluation criteria.