Dynamically altering agent behaviors using natural language instructions
AGENTS '00 Proceedings of the fourth international conference on Autonomous agents
COSIT '99 Proceedings of the International Conference on Spatial Information Theory: Cognitive and Computational Foundations of Geographic Information Science
Understanding natural language instructions: the case of purpose clauses
ACL '92 Proceedings of the 30th annual meeting on Association for Computational Linguistics
COLING '02 Proceedings of the 19th international conference on Computational linguistics - Volume 1
Semiotic schemas: a framework for grounding language in action and perception
Artificial Intelligence - Special volume on connecting language to the world
NLTK: the natural language toolkit
ACLdemo '04 Proceedings of the ACL 2004 on Interactive poster and demonstration sessions
Wayfinding choremes-a language for modeling conceptual route knowledge
Journal of Visual Languages and Computing
Spatial language for human-robot dialogs
IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews
AAAI'07 Proceedings of the 22nd national conference on Artificial intelligence - Volume 2
Where to go: interpreting natural directions using global inference
ICRA'09 Proceedings of the 2009 IEEE international conference on Robotics and Automation
Following directions using statistical machine translation
Proceedings of the 5th ACM/IEEE international conference on Human-robot interaction
Toward understanding natural language directions
Proceedings of the 5th ACM/IEEE international conference on Human-robot interaction
Factoring the Mapping Problem: Mobile Robot Map-building in the Hybrid Spatial Semantic Hierarchy
International Journal of Robotics Research
Reading between the lines: learning to map high-level instructions to commands
ACL '10 Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics
Comparing spoken language route instructions for robots across environment representations
SIGDIAL '10 Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Fast online lexicon learning for grounded language acquisition
ACL '12 Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1
Corpus-based interpretation of instructions in virtual environments
ACL '12 Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers - Volume 2
Toward learning perceptually grounded word meanings from unaligned parallel data
SIAC '12 Proceedings of the Second Workshop on Semantic Interpretation in an Actionable Context
Unsupervised PCFG induction for grounded language learning with highly ambiguous supervision
EMNLP-CoNLL '12 Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning
Landmark-based location belief tracking in a spoken dialog system
SIGDIAL '12 Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Interacting with a robot: a guide robot understanding natural language instructions
UCAmI'12 Proceedings of the 6th international conference on Ubiquitous Computing and Ambient Intelligence
A qualitative path planner for robot navigation using human-provided maps
International Journal of Robotics Research
Hi-index | 0.00 |
Following verbal route instructions requires knowledge of language, space, action and perception. We present MARCO, an agent that follows free-form, natural language route instructions by representing and executing a sequence of compound action specifications that model which actions to take under which conditions. MARCO infers implicit actions from knowledge of both linguistic conditional phrases and from spatial action and local configurations. Thus, MARCO performs explicit actions, implicit actions necessary to achieve the stated conditions, and exploratory actions to learn about the world. We gathered a corpus of 786 route instructions from six people in three large-scale virtual indoor environments. Thirtysix other people followed these instructions and rated them for quality. These human participants finished at the intended destination on 69% of the trials. MARCO followed the same instructions in the same environments, with a success rate of 61%. We measured the efficacy of action inference with MARCO variants lacking action inference: executing only explicit actions, MARCO succeeded on just 28% of the trials. For this task, inferring implicit actions is essential to follow poor instructions, but is also crucial for many highly-rated route instructions.