BLEU: a method for automatic evaluation of machine translation
ACL '02 Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Intrinsic vs. extrinsic evaluation measures for referring expression generation
HLT-Short '08 Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers
The TUNA challenge 2008: overview and evaluation results
INLG '08 Proceedings of the Fifth International Natural Language Generation Conference
The prevalence of descriptive referring expressions in news and narrative
ACLShort '10 Proceedings of the ACL 2010 Conference Short Papers
Comparing rating scales and preference judgements in language evaluation
INLG '10 Proceedings of the 6th International Natural Language Generation Conference
Charting the potential of description logic for the generation of referring expressions
INLG '10 Proceedings of the 6th International Natural Language Generation Conference
Towards an extrinsic evaluation of referring expressions in situated dialogs
INLG '10 Proceedings of the 6th International Natural Language Generation Conference
Introducing shared tasks to NLG: the TUNA shared task evaluation challenges
Empirical methods in natural language generation
The first challenge on generating instructions in virtual environments
Empirical methods in natural language generation
What computational linguists can learn from psychologists (and vice versa)
Computational Linguistics
Does size matter: how much data is required to train a REG algorithm?
HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume 2
Computational generation of referring expressions: A survey
Computational Linguistics
The impact of visual context on the content of referring expressions
ENLG '11 Proceedings of the 13th European Workshop on Natural Language Generation
GRE3D7: a corpus of distinguishing descriptions for objects in visual scenes
UCNLG+EVAL '11 Proceedings of the UCNLG+Eval: Language Generation and Evaluation Workshop
Offline sentence processing measures for testing readability with users
PITR '12 Proceedings of the First Workshop on Predicting and Improving Text Readability for target reader populations
Learning preferences for referring expression generation: effects of domain, language and algorithm
INLG '12 Proceedings of the Seventh International Natural Language Generation Conference
REX-J: Japanese referring expression corpus of situated dialogs
Language Resources and Evaluation
Information Processing and Management: an International Journal
Hi-index | 0.01 |
The TUNA-REG'09 Challenge was one of the shared-task evaluation competitions at Generation Challenges 2009. TUNA-REG'09 used data from the TUNA Corpus of paired representations of entities and human-authored referring expressions. The shared task was to create systems that generate referring expressions for entities given representations of sets of entities and their properties. Four teams submitted six systems to TUNA-REG'09. We evaluated the six systems and two sets of human-authored referring expressions using several automatic intrinsic measures, a human-assessed intrinsic evaluation and a human task performance experiment. This report describes the TUNA-REG task and the evaluation methods used, and presents the evaluation results.