A semi-automatic evaluation scheme: automated nuggetization for manual annotation

Authors:
Liang Zhou;Namhee Kwon;Eduard Hovy
Affiliations:
University of Southern California, Marina del Key, CA;University of Southern California, Marina del Key, CA;University of Southern California, Marina del Key, CA
Venue:
NAACL-Short '07 Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers
Year:
2007

Citing 4
Cited 1

Head-driven statistical models for natural language parsing

Head-driven statistical models for natural language parsing
Automatic evaluation of summaries using N-gram co-occurrence statistics

NAACL '03 Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1
Automatically evaluating answers to definition questions

HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing
Nuggeteer: automatic nugget-based evaluation using descriptions and judgements

HLT-NAACL '06 Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics

Heuristic algorithm for extraction of facts using relational model and syntactic data

MICAI'11 Proceedings of the 10th Mexican international conference on Advances in Artificial Intelligence - Volume Part I

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this paper we describe automatic information nuggetization and its application to text comparison. More specifically, we take a close look at how machine-generated nuggets can be used to create evaluation material. A semiautomatic annotation scheme is designed to produce gold-standard data with exceptionally high inter-human agreement.