The Role of Occam‘s Razor in Knowledge Discovery

  • Authors:
  • Pedro Domingos

  • Affiliations:
  • Department of Computer Science and Engineering, University of Washington, Seattle, WA 98195. pedrod@cs.washington.edu

  • Venue:
  • Data Mining and Knowledge Discovery
  • Year:
  • 1999

Quantified Score

Hi-index 0.02

Visualization

Abstract

Many KDD systems incorporate an implicit or explicitpreference for simpler models, but this use of “Occam‘s razor” hasbeen strongly criticized by several authors (e.g., Schaffer, 1993;Webb, 1996). This controversy arises partly because Occam‘s razor hasbeen interpreted in two quite different ways. The firstinterpretation (simplicity is a goal in itself) is essentiallycorrect, but is at heart a preference for more comprehensible models.The second interpretation (simplicity leads to greater accuracy) ismuch more problematic. A critical review of the theoretical argumentsfor and against it shows that it is unfounded as a universalprinciple, and demonstrably false. A review of empirical evidenceshows that it also fails as a practical heuristic. This articleargues that its continued use in KDD risks causing significantopportunities to be missed, and should therefore be restricted to thecomparatively few applications where it is appropriate. The articleproposes and reviews the use of domain constraints as an alternativefor avoiding overfitting, and examines possible methods for handlingthe accuracy–comprehensibility trade-off.