Generating estimates of classification confidence for a case-based spam filter

  • Authors:
  • Sarah Jane Delany;Pádraig Cunningham;Dónal Doyle;Anton Zamolotskikh

  • Affiliations:
  • Dublin Institute of Technology, Dublin 8, Ireland;Trinity College, University of Dublin, Dublin 2, Ireland;Trinity College, University of Dublin, Dublin 2, Ireland;Trinity College, University of Dublin, Dublin 2, Ireland

  • Venue:
  • ICCBR'05 Proceedings of the 6th international conference on Case-Based Reasoning Research and Development
  • Year:
  • 2005

Quantified Score

Hi-index 0.00

Visualization

Abstract

Producing estimates of classification confidence is surprisingly difficult. One might expect that classifiers that can produce numeric classification scores (e.g. k-Nearest Neighbour, Naïve Bayes or Support Vector Machines) could readily produce confidence estimates based on thresholds. In fact, this proves not to be the case, probably because these are not probabilistic classifiers in the strict sense. The numeric scores coming from k-Nearest Neighbour, Naïve Bayes and Support Vector Machine classifiers are not well correlated with classification confidence. In this paper we describe a case-based spam filtering application that would benefit significantly from an ability to attach confidence predictions to positive classifications (i.e. messages classified as spam). We show that ‘obvious' confidence metrics for a case-based classifier are not effective. We propose an ensemble-like solution that aggregates a collection of confidence metrics and show that this offers an effective solution in this spam filtering domain.