ROC curve equivalence using the Kolmogorov-Smirnov test

Authors:
Andrew P. Bradley
Affiliations:
The University of Queensland, School of Information Technology and Electrical Engineering, St. Lucia, QLD 4072, Australia
Venue:
Pattern Recognition Letters
Year:
2013

Citing 7
Cited 1

ROC curves and the X2 test

Pattern Recognition Letters
Sample Size Estimation using the Receiver Operating Characteristic Curve

ICPR '04 Proceedings of the Pattern Recognition, 17th International Conference on (ICPR'04) Volume 4 - Volume 04
An introduction to ROC analysis

Pattern Recognition Letters - Special issue: ROC analysis in pattern recognition
Cost curves: An improved method for visualizing classifier performance

Machine Learning
Precision-recall operating characteristic (P-ROC) curves in imprecise environments

ICPR '06 Proceedings of the 18th International Conference on Pattern Recognition - Volume 04
Measuring classifier performance: a coherent alternative to the area under the ROC curve

Machine Learning
The use of the area under the ROC curve in the evaluation of machine learning algorithms

Pattern Recognition

Half-AUC for the evaluation of sensitive or specific classifiers

Pattern Recognition Letters

Quantified Score

Hi-index	0.10

Visualization

Abstract

This paper describes a simple, non-parametric and generic test of the equivalence of receiver operating characteristic (ROC) curves based on a modified Kolmogorov-Smirnov (KS) test. The test is described in relation to the commonly used techniques such as the area under the ROC curve (AUC) and the Neyman-Pearson method. We first review how the KS test is used to test the null hypotheses that the class labels predicted by a classifier are no better than random. We then propose an interval mapping technique that allows us to use two KS tests to test the null hypothesis that two classifiers have ROC curves that are equivalent. We demonstrate that this test discriminates different ROC curves both when one curve dominates another and when the curves cross and so are not discriminated by AUC. The interval mapping technique is then used to demonstrate that, although AUC has its limitations, it can be a model-independent and coherent measure of classifier performance.