Definitional, personal, and mechanical constraints on part of speech annotation performance

Authors:
Anna Babarczy;John Carroll;Geoffrey Sampson
Affiliations:
Department of Informatics, University of Sussex, Falmer, Brighton BN1 9QH, UK;Department of Informatics, University of Sussex, Falmer, Brighton BN1 9QH, UK;Department of Informatics, University of Sussex, Falmer, Brighton BN1 9QH, UK
Venue:
Natural Language Engineering
Year:
2006

Citing 6
Cited 2

Assessing agreement on classification tasks: the kappa statistic

Computational Linguistics
Does Baum-Welch re-estimation help taggers?

ANLC '94 Proceedings of the fourth conference on Applied natural language processing
An experiment on the upper bound of interjudge agreement: the case of tagging

EACL '99 Proceedings of the ninth conference on European chapter of the Association for Computational Linguistics
CLAWS4: the tagging of the British National Corpus

COLING '94 Proceedings of the 15th conference on Computational linguistics - Volume 1
Detecting errors in part-of-speech annotation

EACL '03 Proceedings of the tenth conference on European chapter of the Association for Computational Linguistics - Volume 1
Handling noisy training and testing data

EMNLP '02 Proceedings of the ACL-02 conference on Empirical methods in natural language processing - Volume 10

Inter-coder agreement for computational linguistics

Computational Linguistics
Definitional and human constraints on structural annotation of english*

Natural Language Engineering

Quantified Score

Hi-index	0.00

Visualization

Abstract

For one aspect of grammatical annotation, part-of-speech tagging, we investigate experimentally whether the ceiling on accuracy stems from limits to the precision of tag definition or limits to analysts' ability to apply precise definitions, and we examine how analysts' performance is affected by alternative types of semi-automatic support. We find that, even for analysts very well-versed in a part-of-speech tagging scheme, human ability to conform to the scheme is a more serious constraint than precision of scheme definition. We also find that although semi-automatic techniques can greatly increase speed relative to manual tagging, they have little effect on accuracy, either positively (by suggesting valid candidate tags) or negatively (by lending an appearance of authority to incorrect tag assignments). On the other hand, it emerges that there are large differences between individual analysts with respect to usability of particular types of semi-automatic support.