Identifying sources of inter-annotator variation: evaluating two models of argument analysis

Authors:
Barbara White
Affiliations:
The University of Western Ontario, London, ON, Canada
Venue:
LAW IV '10 Proceedings of the Fourth Linguistic Annotation Workshop
Year:
2010

Citing 3
Cited 1

An annotation scheme for discourse-level argumentation in research articles

EACL '99 Proceedings of the ninth conference on European chapter of the Association for Computational Linguistics
Reliability measurement without limits

Computational Linguistics
Annotating a corpus of biomedical research texts: two models of rhetorical analysis

Annotating a corpus of biomedical research texts: two models of rhetorical analysis

Book review: the structure of scientific articles: Applications to citation indexing and summarization simone teufel (university of cambridge) stanford, ca: csli publications (csli studies in computational linguistics), 2010, xii+518 pp; hardbound, isbn 978-1-57586-555-3, $70.00; paperbound, isbn 978-1-57586-556-0, $32.50

Computational Linguistics

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper reports on a pilot study where two Models of argument were applied to the Discussion sections of a corpus of biomedical research articles. The goal was to identify sources of systematic inter-annotator variation as diagnostics for improving the Models. In addition to showing a need to revise both Models, the results identified problems resulting from limitations in annotator expertise. In future work two types of annotators are required: those with biomedical domain expertise and those with an understanding of rhetorical structure.