Exploiting 'subjective' annotations

Authors:
Dennis Reidsma;Rieks op den Akker
Affiliations:
University of Twente, Enschede, The Netherlands;University of Twente, Enschede, The Netherlands
Venue:
HumanJudge '08 Proceedings of the Workshop on Human Judgements in Computational Linguistics
Year:
2008

Citing 9
Cited 8

Variations in relevance judgments and the measurement of retrieval effectiveness

Information Processing and Management: an International Journal
Development and use of a gold-standard data set for subjectivity classifications

ACL '99 Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics
Evaluating Discourse and Dialogue Coding Schemes

Computational Linguistics
Identifying agreement and disagreement in conversational speech: use of Bayesian networks to model pragmatic dependencies

ACL '04 Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics
Reliability measurement without limits

Computational Linguistics
The reliability of anaphoric annotation, reconsidered: taking ambiguity into account

CorpusAnno '05 Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky
Computing backchannel distributions in multi-party conversations

EmbodiedNLP '07 Proceedings of the Workshop on Embodied Language Processing
Challenges for virtual humans in human computing

ICMI'06/IJCAI'07 Proceedings of the ICMI 2006 and IJCAI 2007 international conference on Artifical intelligence for human computing
A study on visual focus of attention recognition from head pose in a meeting room

MLMI'06 Proceedings of the Third international conference on Machine Learning for Multimodal Interaction

That's nice... what can you do with it?

Computational Linguistics
From annotator agreement to noise models

Computational Linguistics
Learning with annotation noise

ACL '09 Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - Volume 1
Towards a better understanding of uncertainties and speculations in Swedish clinical text: analysis of an initial annotation trial

NeSp-NLP '10 Proceedings of the Workshop on Negation and Speculation in Natural Language Processing
Levels of certainty in knowledge-intensive corpora: an initial annotation study

NeSp-NLP '10 Proceedings of the Workshop on Negation and Speculation in Natural Language Processing
Assessing the trade-off between system building cost and output quality in data-to-text generation

Empirical methods in natural language generation
Evaluating the visual quality of web pages using a computational aesthetic approach

Proceedings of the fourth ACM international conference on Web search and data mining
Aggregation of multiple judgments for evaluating ordered lists

ECIR'2010 Proceedings of the 32nd European conference on Advances in Information Retrieval

Quantified Score

Hi-index	0.00

Visualization

Abstract

Many interesting phenomena in conversation can only be annotated as a subjective task, requiring interpretative judgements from annotators. This leads to data which is annotated with lower levels of agreement not only due to errors in the annotation, but also due to the differences in how annotators interpret conversations. This paper constitutes an attempt to find out how subjective annotations with a low level of agreement can profitably be used for machine learning purposes. We analyse the (dis)agreements between annotators for two different cases in a multimodal annotated corpus and explicitly relate the results to the way machine-learning algorithms perform on the annotated data. Finally we present two new concepts, namely 'subjective entity' classifiers resp. 'consensus objective' classifiers, and give recommendations for using subjective data in machine-learning applications.