The first challenge on generating instructions in virtual environments

Authors:
Alexander Koller;Kristina Striegnitz;Donna Byron;Justine Cassell;Robert Dale;Johanna Moore;Jon Oberlander
Affiliations:
Saarland University, Saarbrücken, Germany;Union College, Schenectady, NY;Northeastern University, Boston, MA;Northwestern University, Evanston, IL;Macquarie University, Sydney, Australia;University of Edinburgh, Edinburgh, UK;University of Edinburgh, Edinburgh, UK
Venue:
Empirical methods in natural language generation
Year:
2010

Citing 14
Cited 16

PARADISE: a framework for evaluating spoken dialogue agents

ACL '98 Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
Labeling images with a computer game

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Evaluation metrics for generation

INLG '00 Proceedings of the first international conference on Natural language generation - Volume 14
That's nice... what can you do with it?

Computational Linguistics
Intrinsic vs. extrinsic evaluation measures for referring expression generation

HLT-Short '08 Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers
The TUNA-REG Challenge 2009: overview and evaluation results

ENLG '09 Proceedings of the 12th European Workshop on Natural Language Generation
Addressing the resource bottleneck to create large-scale annotated texts

STEP '08 Proceedings of the 2008 Conference on Semantics in Text Processing
Noun phrase generation for situated dialogs

INLG '06 Proceedings of the Fourth International Natural Language Generation Conference
Automated metrics that agree with human judgements on generated output for an embodied conversational agent

INLG '08 Proceedings of the Fifth International Natural Language Generation Conference
Assessing the trade-off between system building cost and output quality in data-to-text generation

Empirical methods in natural language generation
Human evaluation of a german surface realisation ranker

Empirical methods in natural language generation
Introducing shared tasks to NLG: the TUNA shared task evaluation challenges

Empirical methods in natural language generation
Generating referring expressions in context: the GREC task evaluation challenges

Empirical methods in natural language generation
Evaluating evaluation methods for generation in the presence of variation

CICLing'05 Proceedings of the 6th international conference on Computational Linguistics and Intelligent Text Processing

Automated planning for situated natural language generation

ACL '10 Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics
Report on the second NLG challenge on generating instructions in virtual environments (GIVE-2)

INLG '10 Proceedings of the 6th International Natural Language Generation Conference
Introducing shared tasks to NLG: the TUNA shared task evaluation challenges

Empirical methods in natural language generation
Generating referring expressions in context: the GREC task evaluation challenges

Empirical methods in natural language generation
Hierarchical reinforcement learning and hidden Markov models for task-oriented natural language generation

HLT '11 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers - Volume 2
Modeling spatial knowledge for generating verbal and visual route directions

KES'11 Proceedings of the 15th international conference on Knowledge-based and intelligent information and engineering systems - Volume Part IV
Computational generation of referring expressions: A survey

Computational Linguistics
Optimising natural language generation decision making for situated dialogue

SIGDIAL '11 Proceedings of the SIGDIAL 2011 Conference
Combining hierarchical reinforcement learning and Bayesian networks for natural language generation in situated dialogue

ENLG '11 Proceedings of the 13th European Workshop on Natural Language Generation
Combining symbolic and corpus-based approaches for the generation of successful referring expressions

ENLG '11 Proceedings of the 13th European Workshop on Natural Language Generation
Report on the second second challenge on generating instructions in virtual environments (GIVE-2.5)

ENLG '11 Proceedings of the 13th European Workshop on Natural Language Generation
Direction giving: an attempt to increase user engagement

ENLG '11 Proceedings of the 13th European Workshop on Natural Language Generation
The Bremen system for the GIVE-2.5 challenge

ENLG '11 Proceedings of the 13th European Workshop on Natural Language Generation
The Potsdam NLG systems at the GIVE-2.5 challenge

ENLG '11 Proceedings of the 13th European Workshop on Natural Language Generation
Generation of landmark-based navigation instructions from open-source data

EACL '12 Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics
Enhancing referential success by tracking hearer gaze

SIGDIAL '12 Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper describes the First Challenge on Generating Instructions in Virtual Environments (GIVE-1). GIVE is a shared task for generation systems which give real-time natural-language instructions to users in a virtual 3D world. These systems are evaluated by connecting users and NLG systems over the Internet. We describe the design and results of GIVE-1 as well as the participating NLG systems, and validate the experimental methodology by comparing the results collected over the Internet with results from a more traditional laboratory-based experiment.