Implications of ceiling effects in defect predictors

  • Authors:

  • Tim Menzies;Burak Turhan;Ayse Bener;Gregory Gay;Bojan Cukic;Yue Jiang

  • Affiliations:

  • CSEE, WVU, Morgantown, WV, USA;Dept. of Computer Engineering, Bogazici, Turkey;Dept. of Computer Engineering, Bogazici, Turkey;LCSEE, WVU, Morgantown, USA;LCSEE, WVU, Morgantown, USA;LCSEE, WVU, Morgantown , USA

  • Venue:
  • Proceedings of the 4th international workshop on Predictor models in software engineering
  • Year:
  • 2008

Quantified Score

Hi-index 0.00

Visualization

Abstract

Context: There are many methods that input static code features and output a predictor for faulty code modules. These data mining methods have hit a "performance ceiling"; i.e., some inherent upper bound on the amount of information offered by, say, static code features when identifying modules which contain faults. Objective: We seek an explanation for this ceiling effect. Perhaps static code features have "limited information content"; i.e. their information can be quickly and completely discovered by even simple learners. Method:An initial literature review documents the ceiling effect in other work. Next, using three sub-sampling techniques (under-, over-, and micro-sampling), we look for the lower useful bound on the number of training instances. Results: Using micro-sampling, we find that as few as 50 instances yield as much information as larger training sets. Conclusions: We have found much evidence for the limited information hypothesis. Further progress in learning defect predictors may not come from better algorithms. Rather, we need to be improving the information content of the training data, perhaps with case-based reasoning methods.