Augmenting the kappa statistic to determine interannotator reliability for multiply labeled data points

Authors:
Andrew Rosenberg;Ed Binkowski
Affiliations:
Columbia University;Hunter College
Venue:
HLT-NAACL-Short '04 Proceedings of HLT-NAACL 2004: Short Papers
Year:
2004

Citing 0
Cited 7

Inter-coder agreement for computational linguistics

Computational Linguistics
The reliability of anaphoric annotation, reconsidered: taking ambiguity into account

CorpusAnno '05 Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky
An agreement measure for determining inter-annotator reliability of human judgements on affective text

HumanJudge '08 Proceedings of the Workshop on Human Judgements in Computational Linguistics
How useful are your comments?: analyzing and predicting youtube comments and comment ratings

Proceedings of the 19th international conference on World wide web
Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge

Speech Communication
Detection of spam tipping behaviour on foursquare

Proceedings of the 22nd international conference on World Wide Web companion
Software testing with an operational profile: OP definition

ACM Computing Surveys (CSUR)

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper describes a method for evaluating interannotator reliability in an email corpus annotated for type (e.g., question, answer, social chat) when annotators are allowed to assign multiple labels to a message. An augmentation is proposed to Cohen's kappa statistic which permits all data to be included in the reliability measure and which further permits the identification of more or less reliably annotated data points.