On the effect of calibration in classifier combination

Authors:
Antonio Bella;Cèsar Ferri;José Hernández-Orallo;María José Ramírez-Quintana
Affiliations:
DSIC-ELP, Universitat Politècnica de València, Valencia, Spain 46022;DSIC-ELP, Universitat Politècnica de València, Valencia, Spain 46022;DSIC-ELP, Universitat Politècnica de València, Valencia, Spain 46022;DSIC-ELP, Universitat Politècnica de València, Valencia, Spain 46022
Venue:
Applied Intelligence
Year:
2013

Citing 28
Cited 1

Original Contribution: Stacked generalization

Neural Networks
Bagging predictors

Machine Learning
An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization

Machine Learning
Cascade Generalization

Machine Learning
A Theoretical Study on Six Classifier Fusion Strategies

IEEE Transactions on Pattern Analysis and Machine Intelligence
Data mining: practical machine learning tools and techniques with Java implementations

ACM SIGMOD Record
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems

Machine Learning
Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy

Machine Learning
Ensemble Methods in Machine Learning

MCS '00 Proceedings of the First International Workshop on Multiple Classifier Systems
Transforming classifier scores into accurate multiclass probability estimates

Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining
Combining Pattern Classifiers: Methods and Algorithms

Combining Pattern Classifiers: Methods and Algorithms
In Defense of One-Vs-All Classification

The Journal of Machine Learning Research
Data mining in metric space: an empirical analysis of supervised learning performance criteria

Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining
Delegating classifiers

ICML '04 Proceedings of the twenty-first international conference on Machine learning
The Combination of Text Classifiers Using Reliability Indicators

Information Retrieval
Properties and benefits of calibrated classifiers

PKDD '04 Proceedings of the 8th European Conference on Principles and Practice of Knowledge Discovery in Databases
Getting the Most Out of Ensemble Selection

ICDM '06 Proceedings of the Sixth International Conference on Data Mining
Building reliable metaclassifiers for text learning

Building reliable metaclassifiers for text learning
Statistical Comparisons of Classifiers over Multiple Data Sets

The Journal of Machine Learning Research
An experimental comparison of performance measures for classification

Pattern Recognition Letters
Similarity-binning averaging: a generalisation of binning calibration

IDEAL'09 Proceedings of the 10th international conference on Intelligent data engineering and automated learning
A low variance error boosting algorithm

Applied Intelligence
Applying correlation to enhance boosting technique using genetic programming as base learner

Applied Intelligence
Hybrid ensemble approach for classification

Applied Intelligence
Random projections for linear SVM ensembles

Applied Intelligence
Ensembles of ARTMAP-based neural networks: an experimental study

Applied Intelligence
A cascaded classifier approach for improving detection rates on rare attack categories in network intrusion detection

Applied Intelligence
EEM: evolutionary ensembles model for activity recognition in Smart Homes

Applied Intelligence

Aggregative quantification for regression

Data Mining and Knowledge Discovery

Quantified Score

Hi-index	0.00

Visualization

Abstract

A general approach to classifier combination considers each model as a probabilistic classifier which outputs a class membership posterior probability. In this general scenario, it is not only the quality and diversity of the models which are relevant, but the level of calibration of their estimated probabilities as well. In this paper, we study the role of calibration before and after classifier combination, focusing on evaluation measures such as MSE and AUC, which better account for good probability estimation than other evaluation measures. We present a series of findings that allow us to recommend several layouts for the use of calibration in classifier combination. We also empirically analyse a new non-monotonic calibration method that obtains better results for classifier combination than other monotonic calibration methods.