Transforming classifier scores into accurate multiclass probability estimates

Authors:
Bianca Zadrozny;Charles Elkan
Affiliations:
University of California, San Diego, La Jolla, California;University of California, San Diego, La Jolla, California
Venue:
Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining
Year:
2002

Citing 6
Cited 76

Classification by pairwise coupling

NIPS '97 Proceedings of the 1997 conference on Advances in neural information processing systems 10
Learning and making decisions when costs and probabilities are both unknown

Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining
Magical thinking in data mining: lessons from CoIL challenge 2000

Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining
Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers

ICML '01 Proceedings of the Eighteenth International Conference on Machine Learning
Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers

ICML '00 Proceedings of the Seventeenth International Conference on Machine Learning
Solving multiclass learning problems via error-correcting output codes

Journal of Artificial Intelligence Research

Exploiting Unlabeled Data for Improving Accuracy of Predictive Data Mining

ICDM '03 Proceedings of the Third IEEE International Conference on Data Mining
A Bayesian network framework for reject inference

Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining
Probabilistic score estimation with piecewise logistic regression

ICML '04 Proceedings of the twenty-first international conference on Machine learning
Local sparsity control for naive Bayes with extreme misclassification costs

Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining
One-Benefit learning: cost-sensitive learning with restricted cost information

UBDM '05 Proceedings of the 1st international workshop on Utility-based data mining
Error limiting reductions between classification tasks

ICML '05 Proceedings of the 22nd international conference on Machine learning
Predicting good probabilities with supervised learning

ICML '05 Proceedings of the 22nd international conference on Machine learning
Augmenting naive Bayes for ranking

ICML '05 Proceedings of the 22nd international conference on Machine learning
Effective Estimation of Posterior Probabilities: Explaining the Accuracy of Randomized Decision Tree Approaches

ICDM '05 Proceedings of the Fifth IEEE International Conference on Data Mining
Predicting Protein-Protein Interactions by Association Mining

Information Systems Frontiers
An empirical comparison of supervised learning algorithms

ICML '06 Proceedings of the 23rd international conference on Machine learning
Confidence-Based Active Learning

IEEE Transactions on Pattern Analysis and Machine Intelligence
Estimating class priors in domain adaptation for word sense disambiguation

ACL-44 Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics
Confidence-based classifier design

Pattern Recognition
Minimax Regret Classifier for Imprecise Class Distributions

The Journal of Machine Learning Research
PAV and the ROC convex hull

Machine Learning
Mining data from intensive care patients

Advanced Engineering Informatics
Making generative classifiers robust to selection bias

Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining
Methods for person identification on a pressure-sensitive floor: Experiments with multiple classifiers and reject option

Information Fusion
Reliability estimation of a statistical classifier

Pattern Recognition Letters
Enhanced hierarchical classification via isotonic smoothing

Proceedings of the 17th international conference on World Wide Web
An empirical evaluation of supervised learning in high dimensions

Proceedings of the 25th international conference on Machine learning
Cost-sensitive multi-class classification from probability estimates

Proceedings of the 25th international conference on Machine learning
Learning classifiers from only positive and unlabeled data

Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining
Probability Based Metrics for Locally Weighted Naive Bayes

CAI '07 Proceedings of the 20th conference of the Canadian Society for Computational Studies of Intelligence on Advances in Artificial Intelligence
On Pairwise Naive Bayes Classifiers

ECML '07 Proceedings of the 18th European conference on Machine Learning
Semi-Supervised Boosting for Multi-Class Classification

ECML PKDD '08 Proceedings of the European conference on Machine Learning and Knowledge Discovery in Databases - Part II
Naive Bayes for optimal ranking

Journal of Experimental & Theoretical Artificial Intelligence
Calibrated lazy associative classification

SBBD '08 Proceedings of the 23rd Brazilian symposium on Databases
A closed-form reduction of multi-class cost-sensitive learning to weighted multi-class learning

Pattern Recognition
Large scale multi-label classification via metalabeler

Proceedings of the 18th international conference on World wide web
An application of methods for the probabilistic three-class classification of pregnancies of unknown location

Artificial Intelligence in Medicine
Combining audio content and social context for semantic music discovery

Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval
Combining predictions in pairwise classification: An optimal adaptive voting strategy and its relation to weighted voting

Pattern Recognition
Cost-sensitive learning based on Bregman divergences

Machine Learning
Learning with probabilistic features for improved pipeline models

EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Score calibrating for speaker recognition based on support vector machines and Gaussian Mixture Models

SIP '07 Proceedings of the Ninth IASTED International Conference on Signal and Image Processing
A Study of Parts-Based Object Class Detection Using Complete Graphs

International Journal of Computer Vision
A regularization framework for multiclass classification: A deterministic annealing approach

Pattern Recognition
Risk neutral calibration of classifiers

AI'03 Proceedings of the 16th Canadian society for computational studies of intelligence conference on Advances in artificial intelligence
Estimation of class membership probabilities in the document classification

PAKDD'07 Proceedings of the 11th Pacific-Asia conference on Advances in knowledge discovery and data mining
A large-scale active learning system for topical categorization on the web

Proceedings of the 19th international conference on World wide web
Comparing methods for multi-class probabilities in medical decision making using LS-SVMs and kernel logistic regression

ICANN'07 Proceedings of the 17th international conference on Artificial neural networks
Combining clauses with various precisions and recalls to produce accurate probabilistic estimates

ILP'07 Proceedings of the 17th international conference on Inductive logic programming
Handling over-fitting in test cost-sensitive decision tree learning by feature selection, smoothing and pruning

Journal of Systems and Software
Cold start link prediction

Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining
SVM-FuzCoC: A novel SVM-based feature selection method using a fuzzy complementary criterion

Pattern Recognition
A decision support system for cost-effective diagnosis

Artificial Intelligence in Medicine
Uncertainty in clustering and classification

SUM'10 Proceedings of the 4th international conference on Scalable uncertainty management
Calibrated lazy associative classification

Information Sciences: an International Journal
Categorization of display ads using image and landing page features

Proceedings of the Third Workshop on Large Scale Data Mining: Theory and Applications
Learning multi-class theories in ILP

ILP'10 Proceedings of the 20th international conference on Inductive logic programming
Bayesian classifiers for positive unlabeled learning

WAIM'11 Proceedings of the 12th international conference on Web-age information management
Accurate information extraction for quantitative financial events

Proceedings of the 20th ACM international conference on Information and knowledge management
Effective probability forecasting for time series data using standard machine learning techniques

ICAPR'05 Proceedings of the Third international conference on Advances in Pattern Recognition - Volume Part I
Robust probabilistic calibration

ECML'06 Proceedings of the 17th European conference on Machine Learning
StackTIS: A stacked generalization approach for effective prediction of translation initiation sites

Computers in Biology and Medicine
Attribute and object selection queries on objects with probabilistic attributes

ACM Transactions on Database Systems (TODS)
Effective confidence region prediction using probability forecasters

AIME'05 Proceedings of the 10th conference on Artificial Intelligence in Medicine
Discriminative vs. generative classifiers for cost sensitive learning

AI'06 Proceedings of the 19th international conference on Advances in Artificial Intelligence: Canadian Society for Computational Studies of Intelligence
Effects of data grouping on calibration measures of classifiers

EUROCAST'11 Proceedings of the 13th international conference on Computer Aided Systems Theory - Volume Part I
Understanding web images by object relation network

Proceedings of the 21st international conference on World Wide Web
Probabilistic outputs for twin support vector machines

Knowledge-Based Systems
Fuzzy machine learning and data mininga

Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery
Georeferencing Flickr photos using language models at different levels of granularity: An evidence based approach

Web Semantics: Science, Services and Agents on the World Wide Web
Learning and inference in probabilistic classifier chains with beam search

ECML PKDD'12 Proceedings of the 2012 European conference on Machine Learning and Knowledge Discovery in Databases - Volume Part I
Probability estimation for multi-class classification based on label ranking

ECML PKDD'12 Proceedings of the 2012 European conference on Machine Learning and Knowledge Discovery in Databases - Volume Part II
A unified view of performance metrics: translating threshold choice into expected classification loss

The Journal of Machine Learning Research
Inferring anchor links across multiple heterogeneous social networks

Proceedings of the 22nd ACM international conference on Conference on information & knowledge management
On the effect of calibration in classifier combination

Applied Intelligence
Beam search algorithms for multilabel learning

Machine Learning
Accurate probability calibration for multiple classifiers

IJCAI'13 Proceedings of the Twenty-Third international joint conference on Artificial Intelligence
Alleviating naive Bayes attribute independence assumption by attribute weighting

The Journal of Machine Learning Research
Learning by abstraction: Hierarchical classification model using evidential theoretic approach and Bayesian ensemble model

Neurocomputing
Aggregative quantification for regression

Data Mining and Knowledge Discovery
Enhancing understanding and improving prediction of severe weather through spatiotemporal relational learning

Machine Learning

Quantified Score

Hi-index	0.00

Visualization

Abstract

Class membership probability estimates are important for many applications of data mining in which classification outputs are combined with other sources of information for decision-making, such as example-dependent misclassification costs, the outputs of other classifiers, or domain knowledge. Previous calibration methods apply only to two-class problems. Here, we show how to obtain accurate probability estimates for multiclass problems by combining calibrated binary probability estimates. We also propose a new method for obtaining calibrated two-class probability estimates that can be applied to any classifier that produces a ranking of examples. Using naive Bayes and support vector machine classifiers, we give experimental results from a variety of two-class and multiclass domains, including direct marketing, text categorization and digit recognition.