An analysis of data sets used to train and validate cost prediction systems

  • Authors:

  • Carolyn Mair;Martin Shepperd;Magne Jørgensen

  • Affiliations:

  • Bournemouth University, United Kingdom;Bournemouth University, United Kingdom;Simula Labs, Norway

  • Venue:
  • PROMISE '05 Proceedings of the 2005 workshop on Predictor models in software engineering
  • Year:
  • 2005

Quantified Score

Hi-index 0.00

Visualization

Abstract

OBJECTIVE - to build up a picture of the nature and type of data sets being used to develop and evaluate different software project effort prediction systems. We believe this to be important since there is a growing body of published work that seeks to assess different prediction approaches.METHOD - we performed an exhaustive search from 1980 onwards from three software engineering journals for research papers that used project data sets to compare cost prediction systems.RESULTS - this identified a total of 50 papers that used, one or more times, a total of 71 unique project data sets. We observed that some of the better known and easily accessible data sets were used repeatedly making them potentially disproportionately influential. Such data sets also tend to be amongst the oldest with potential problems of obsolescence. We also note that only about 60% of all data sets are in the public domain. Finally, extracting relevant information from research papers has been time consuming due to different styles of presentation and levels of contextural information.CONCLUSIONS - first, the community needs to consider the quality and appropriateness of the data set being utilised; not all data sets are equal. Second, we need to assess the way results are presented in order to facilitate meta-analysis and whether a standard protocol would be appropriate.