An agreement measure for determining inter-annotator reliability of human judgements on affective text

Authors:
Plaban Kr. Bhowmick;Pabitra Mitra;Anupam Basu
Affiliations:
Indian Institute of Technology, Kharagpur, India;Indian Institute of Technology, Kharagpur, India;Indian Institute of Technology, Kharagpur, India
Venue:
HumanJudge '08 Proceedings of the Workshop on Human Judgements in Computational Linguistics
Year:
2008

Citing 8
Cited 0

Assessing agreement on classification tasks: the kappa statistic

Computational Linguistics
Automatic Analysis of Facial Expressions: The State of the Art

IEEE Transactions on Pattern Analysis and Machine Intelligence
Measuring agreement in medical informatics reliability studies

Journal of Biomedical Informatics
Dialogue act modeling for automatic tagging and recognition of conversational speech

Computational Linguistics
The reliability of a dialogue structure coding scheme

Computational Linguistics
Recognizing subjectivity: a case study in manual tagging

Natural Language Engineering
Inter-coder agreement for computational linguistics

Computational Linguistics
Augmenting the kappa statistic to determine interannotator reliability for multiply labeled data points

HLT-NAACL-Short '04 Proceedings of HLT-NAACL 2004: Short Papers

Quantified Score

Hi-index	0.00

Visualization

Abstract

An affective text may be judged to belong to multiple affect categories as it may evoke different affects with varying degree of intensity. For affect classification of text, it is often required to annotate text corpus with affect categories. This task is often performed by a number of human judges. This paper presents a new agreement measure inspired by Kappa coefficient to compute inter-annotator reliability when the annotators have freedom to categorize a text into more than one class. The extended reliability coefficient has been applied to measure the quality of an affective text corpus. An analysis of the factors that influence corpus quality has been provided.