Evaluating classifiers: relation between area under the receiver operator characteristic curve and overall accuracy

Authors:
Maciej A. Mazurowski;Georgia D. Tourassi
Affiliations:
Department of Radiology, Duke University Medical Center, Durham, NC;Department of Radiology, Duke University Medical Center, Durham, NC and Department of Electrical and Computer Engineering, University of Louisville, Louisville, KY
Venue:
IJCNN'09 Proceedings of the 2009 international joint conference on Neural Networks
Year:
2009

Citing 12
Cited 1

Small Sample Size Effects in Statistical Pattern Recognition: Recommendations for Practitioners

IEEE Transactions on Pattern Analysis and Machine Intelligence
Introduction to artificial neural systems

Introduction to artificial neural systems
Support-Vector Networks

Machine Learning
On Dimensionality, Sample Size, and Classification Error of Nonparametric Linear Classification Algorithms

IEEE Transactions on Pattern Analysis and Machine Intelligence
Machine Learning

Machine Learning
The Case against Accuracy Estimation for Comparing Induction Algorithms

ICML '98 Proceedings of the Fifteenth International Conference on Machine Learning
Pattern Classification (2nd Edition)

Pattern Classification (2nd Edition)
A General Model for Finite-Sample Effects in Training and Testing of Competing Classifiers

IEEE Transactions on Pattern Analysis and Machine Intelligence
An introduction to ROC analysis

Pattern Recognition Letters - Special issue: ROC analysis in pattern recognition
Multimodality image registration by maximization of quantitative-qualitative measure of mutual information

Pattern Recognition
2008 Special Issue: Training neural network classifiers for medical decision making: The effects of imbalanced datasets on classification performance

Neural Networks
The use of the area under the ROC curve in the evaluation of machine learning algorithms

Pattern Recognition

Estimation of a Priori Decision Threshold for Collocations Extraction: An Empirical Study

International Journal of Information Technology and Web Engineering

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this study, we investigated the relation between two popular classifier performance measures: area under the receiver operator characteristic curve and overall accuracy. We also evaluated the impact of class imbalance and number of examples in test set on this relation. We perform a set of experiments in which we train multiple neural networks and test them in various, well controlled conditions. The experimental results show that given a large and balanced test set, increase in one performance measure is a very good indicator of increase in the other measure. Furthermore increasing the total number of examples, while keeping the positive class prevalence constant generally increases the correlation between the two measures. Our results also indicate that increasing the extent of class imbalance in the test set has a detrimental effect on this correlation.