The reliability of anaphoric annotation, reconsidered: taking ambiguity into account

Authors:
Massimo Poesio;Ron Artstein
Affiliations:
University of Essex, United Kingdom;University of Essex, United Kingdom
Venue:
CorpusAnno '05 Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky
Year:
2005

Citing 5
Cited 10

Foundations of statistical natural language processing

Foundations of statistical natural language processing
Corelex: systematic polysemy and underspecification

Corelex: systematic polysemy and underspecification
On coreferring: coreference in MUC and related annotation schemes

Computational Linguistics
A corpus-based investigation of definite description use

Computational Linguistics
Augmenting the kappa statistic to determine interannotator reliability for multiply labeled data points

HLT-NAACL-Short '04 Proceedings of HLT-NAACL 2004: Short Papers

Inter-coder agreement for computational linguistics

Computational Linguistics
Introduction to Frontiers in Corpus Annotation II: Pie in the Sky

CorpusAnno '05 Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky
Semantically rich human-aided machine annotation

CorpusAnno '05 Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky
Exploiting 'subjective' annotations

HumanJudge '08 Proceedings of the Workshop on Human Judgements in Computational Linguistics
Addressing the resource bottleneck to create large-scale annotated texts

STEP '08 Proceedings of the 2008 Conference on Semantics in Text Processing
Annotation compatibility working group report

LAC '06 Proceedings of the Workshop on Frontiers in Linguistically Annotated Corpora 2006
Methodological Review: Coreference resolution: A review of general methodologies and applications in the clinical domain

Journal of Biomedical Informatics
CoNLL-2011 shared task: modeling unrestricted coreference in OntoNotes

CONLL Shared Task '11 Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task
To what extent does sentence-internal realisation reflect discourse context?: a study on word order

EACL '12 Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics
Multiplicity and word sense: evaluating and learning from multiply labeled word sense annotations

Language Resources and Evaluation

Quantified Score

Hi-index	0.00

Visualization

Abstract

We report the results of a study of the reliability of anaphoric annotation which (i) involved a substantial number of naive subjects, (ii) used Krippendorff's α instead of K to measure agreement, as recently proposed by Passonneau, and (iii) allowed annotators to mark anaphoric expressions as ambiguous.