Optimal mean-square-error calibration of classifier error estimators under Bayesian models

Authors:
Lori A. Dalton;Edward R. Dougherty
Affiliations:
Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA;Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA and Computational Biology Division, Translational Genomics Research Institute, Phoenix, AZ, USA an ...
Venue:
Pattern Recognition
Year:
2012

Citing 10
Cited 1

Multivariate statistical simulation

Multivariate statistical simulation
Decorrelation of the true and estimated classifier errors in high-dimensional settings

EURASIP Journal on Bioinformatics and Systems Biology
On the sampling distribution of resubstitution and leave-one-out error estimators for linear classifiers

Pattern Recognition
Exact correlation between actual and estimated errors in discrete classification

Pattern Recognition Letters
Joint sampling distribution between actual and estimated classification errors for linear discriminant analysis

IEEE Transactions on Information Theory - Special issue on information theory in molecular biology and neuroscience
Application of the Bayesian MMSE estimator for classification error to gene expression microarray data

Bioinformatics
Bayesian Minimum Mean-Square Error Estimation for Classification Error—Part I: Definition and the Bayesian MMSE Error Estimator for Discrete Classification

IEEE Transactions on Signal Processing
Bayesian Minimum Mean-Square Error Estimation for Classification Error—Part II: Linear Classification of Gaussian Models

IEEE Transactions on Signal Processing
Exact representation of the second-order moments for resubstitution and leave-one-out error estimation for linear discriminant analysis in the univariate heteroskedastic Gaussian model

Pattern Recognition
Analytic Study of Performance of Error Estimators for Linear Discriminant Analysis

IEEE Transactions on Signal Processing

Optimal classifiers with minimum expected error within a Bayesian framework-Part I: Discrete and Gaussian models

Pattern Recognition

Quantified Score

Hi-index	0.01

Visualization

Abstract

A recently proposed Bayesian modeling framework for classification facilitates both the analysis and optimization of error estimation performance. The Bayesian error estimator is then defined to have optimal mean-square error performance, but in many situations closed-form representations are unavailable and approximations may not be feasible. To address this, we present a method to optimally calibrate arbitrary error estimators for minimum mean-square error performance within a supposed Bayesian framework. Assuming a fixed sample size, classification rule and error estimation rule, as well as a fixed Bayesian model, the calibration is done by first computing a calibration function that maps error estimates to their optimally calibrated values off-line. Once found, this calibration function may be easily applied to error estimates on the fly whenever the assumptions apply. We demonstrate that calibrated error estimators offer significant improvement in performance relative to classical error estimators under Bayesian models with both linear and non-linear classification rules.