A Formal Proof of a Paradox Associated with Cohen’s Kappa

Authors:
Matthijs J. Warrens
Affiliations:
Leiden University, Institute of Psychology, Unit Methodology and Statistics, P.O. Box 9555, 2300 RB, Leiden, The Netherlands
Venue:
Journal of Classification
Year:
2010

Citing 0
Cited 3

Inequalities between multi-rater kappas

Advances in Data Analysis and Classification
Cohen's linearly weighted kappa is a weighted average

Advances in Data Analysis and Classification
The problem with kappa

EACL '12 Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics

Quantified Score

Hi-index	0.07

Visualization

Abstract

Suppose two judges each classify a group of objects into one of several nominal categories. It has been observed in the literature that, for fixed observed agreement between the judges, Cohen’s kappa penalizes judges with similar marginals compared to judges who produce different marginals. This paper presents a formal proof of this phenomenon.