About the relationship between ROC curves and Cohen's kappa

Authors:
Arie Ben-David
Affiliations:
Management Information Systems, Department of Technology Management, Holon Institute of Technology, 52 Golomb Street, P.O. Box 305, Holon 58102, Israel
Venue:
Engineering Applications of Artificial Intelligence
Year:
2008

Citing 7
Cited 7

A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms

Machine Learning
Robust Classification for Imprecise Environments

Machine Learning
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems

Machine Learning
The Case against Accuracy Estimation for Comparing Induction Algorithms

ICML '98 Proceedings of the Fifteenth International Conference on Machine Learning
Bootstrap Methods for the Cost-Sensitive Evaluation of Classifiers

ICML '00 Proceedings of the Seventeenth International Conference on Machine Learning
Statistical Comparisons of Classifiers over Multiple Data Sets

The Journal of Machine Learning Research
A lot of randomness is hiding in accuracy

Engineering Applications of Artificial Intelligence

The AUK: A simple alternative to the AUC

Engineering Applications of Artificial Intelligence
Towards automatically detecting whether student learning is shallow

ITS'12 Proceedings of the 11th international conference on Intelligent Tutoring Systems
Improving construct validity yields better models of systematic inquiry, even with less information

UMAP'12 Proceedings of the 20th international conference on User Modeling, Adaptation, and Personalization
WTF? detecting students who are conducting inquiry without thinking fastidiously

UMAP'12 Proceedings of the 20th international conference on User Modeling, Adaptation, and Personalization
The problem with kappa

EACL '12 Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics
Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill

User Modeling and User-Adapted Interaction
Beyond cross-domain learning: Multiple-domain nonnegative matrix factorization

Engineering Applications of Artificial Intelligence

Quantified Score

Hi-index	0.00

Visualization

Abstract

Receiver operating characteristic (ROC) curves are very powerful tools for measuring classifiers' accuracy in binary-class problems. However, their usefulness in real-world multi-class problems has not been demonstrated yet. In these frequently occurring multi-class cases, simple accuracy meters that do compensate for random successes, such as the kappa statistic, are needed. ROC curves are two-dimensional graphs. Kappa is a scalar. Each comes from an entirely different discipline. This research investigates whether they do have anything in common. A mathematical formulation that links ROC spaces with the kappa statistic is derived here for the first time. The understanding of how these two accuracy meters relate to each other can assist in a better understanding of their respective pros and cons.