A scaleable automated quality assurance technique for semantic representations and proposition banks

Authors:
K. Bretonnel Cohen;Lawrence E. Hunter;Martha Palmer
Affiliations:
University of Colorado at Boulder;Computational Bioscience Program, U. of Colorado School of Medicine;University of Colorado at Boulder
Venue:
LAW V '11 Proceedings of the 5th Linguistic Annotation Workshop
Year:
2011

Citing 8
Cited 0

Building a large annotated corpus of English: the penn treebank

Computational Linguistics - Special issue on using large corpora: II
Consistency across the hierarchies of the UMLS semantic network and metathesaurus

Journal of Biomedical Informatics - Special issue: Unified medical language system
The Proposition Bank: An Annotated Corpus of Semantic Roles

Computational Linguistics
Inter-coder agreement for computational linguistics

Computational Linguistics
BIOSMILE: adapting semantic role labeling for biomedical verbs: an exponential model coupled with automatically generated template features

BioNLP '06 Proceedings of the Workshop on Linking Natural Language Processing and Biology: Towards Deeper Biological Literature Analysis
Corpus design for biomedical natural language processing

ISMB '05 Proceedings of the ACL-ISMB Workshop on Linking Biological Literature, Ontologies and Databases: Mining Biological Semantics
A semi-automatic method for annotating a biomedical proposition bank

LAC '06 Proceedings of the Workshop on Frontiers in Linguistically Annotated Corpora 2006
Introduction to the CoNLL-2005 shared task: semantic role labeling

CONLL '05 Proceedings of the Ninth Conference on Computational Natural Language Learning

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper presents an evaluation of an automated quality assurance technique for a type of semantic representation known as a predicate argument structure. These representations are crucial to the development of an important class of corpus known as a proposition bank. Previous work (Cohen and Hunter, 2006) proposed and tested an analytical technique based on a simple discovery procedure inspired by classic structural linguistic methodology. Cohen and Hunter applied the technique manually to a small set of representations. Here we test the feasibility of automating the technique, as well as the ability of the technique to scale to a set of semantic representations and to a corpus many times larger than that used by Cohen and Hunter. We conclude that the technique is completely automatable, uncovers missing sense distinctions and other bad semantic representations, and does scale well, performing at an accuracy of 69% for identifying bad representations. We also report on the implications of our findings for the correctness of the semantic representations in PropBank.