Stating with certainty or stating with doubt: intercoder reliability results for manual annotation of epistemically modalized statements

Authors:
Victoria L. Rubin
Affiliations:
University of Western Ontario, London, Ontario, Canada
Venue:
NAACL-Short '07 Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers
Year:
2007

Citing 0
Cited 6

Identifying Firm-Specific Risk Statements in News Articles

PAISI '09 Proceedings of the Pacific Asia Workshop on Intelligence and Security Informatics
Evaluating a meta-knowledge annotation scheme for bio-events

NeSp-NLP '10 Proceedings of the Workshop on Negation and Speculation in Natural Language Processing
Modality and negation: An introduction to the special issue

Computational Linguistics
Are you sure that this happened? assessing the factuality degree of events in text

Computational Linguistics
Did it happen? the pragmatic complexity of veridicality assessment

Computational Linguistics
Modality and negation in simt use of modality and negation in semantically-informed syntactic mt

Computational Linguistics

Quantified Score

Hi-index	0.00

Visualization

Abstract

Texts exhibit subtle yet identifiable modality about writers' estimation of how true each statement is (e.g., definitely true or somewhat true). This study is an analysis of such explicit certainty and doubt markers in epistemically modalized statements for a written news discourse. The study systematically accounts for five levels of writer's certainty (ABSOLUTE, HIGH, MODERATE, LOW CERTAINTY and UNCERTAINTY) in three news pragmatic contexts: perspective, focus, and time. The study concludes that independent coders' perceptions of the boundaries between shades of certainty in epistemically modalized statements are highly subjective and present difficulties for manual annotation and consequent automation for opinion extraction and sentiment analysis. While stricter annotation instructions and longer coder training can improve inter-coder agreement results, it is not entirely clear that a five-level distinction of certainty is preferable to a simplistic distinction between statements with certainty and statements with doubt.