Estimating the uncertainty in the estimated mean area under the ROC curve of a classifier

Authors:
Waleed A. Yousef;Robert F. Wagner;Murray H. Loew
Affiliations:
Electrical and Computer Engineering, George Washington University, 801 22nd Street, N.W. Washington, DC 20052, United States and Center for Devices and Radiological Health (CDRH), Food and Drug Ad ...;Center for Devices and Radiological Health (CDRH), Food and Drug Administration (FDA), 12720 Twinbrook Pkwy, Rockville, MD 20850, United States;Electrical and Computer Engineering, George Washington University, 801 22nd Street, N.W. Washington, DC 20052, United States
Venue:
Pattern Recognition Letters
Year:
2005

Citing 4
Cited 5

Introduction to statistical pattern recognition (2nd ed.)

Introduction to statistical pattern recognition (2nd ed.)
A General Model for Finite-Sample Effects in Training and Testing of Competing Classifiers

IEEE Transactions on Pattern Analysis and Machine Intelligence
Comparison of Non-Parametric Methods for Assessing Classifier Performance in Terms of ROC Parameters

AIPR '04 Proceedings of the 33rd Applied Imagery Pattern Recognition Workshop
The use of the area under the ROC curve in the evaluation of machine learning algorithms

Pattern Recognition

Assessing Classifiers from Two Independent Data Sets Using ROC Analysis: A Nonparametric Approach

IEEE Transactions on Pattern Analysis and Machine Intelligence
2008 Special Issue: Reader studies for validation of CAD systems

Neural Networks
A sorting optimization curve with quality and yield requirements

Pattern Recognition Letters
Uncertainty estimation with a finite dataset in the assessment of classification models

Computational Statistics & Data Analysis
Assessing classifiers in terms of the partial area under the ROC curve

Computational Statistics & Data Analysis

Quantified Score

Hi-index	0.10

Visualization

Abstract

This article considers the problem of binary classification and its assessment in a distribution-free approach. We estimate the area under the ROC curve (a more general performance metric than the error rate) of a classifier using a bootstrap-based estimator. We then use the method of the influence function to estimate the uncertainty of that estimate from the very same bootstrap samples. Monte Carlo trials show that small-sample estimates can be obtained with little bias.