Training a multilingual sportscaster: using perceptual context to learn language

Authors:
David L. Chen;Joohyun Kim;Raymond J. Mooney
Affiliations:
Department of Computer Science, The University of Texas at Austin, Austin, TX;Department of Computer Science, The University of Texas at Austin, Austin, TX;Department of Computer Science, The University of Texas at Austin, Austin, TX
Venue:
Journal of Artificial Intelligence Research
Year:
2010

Citing 43
Cited 9

Discourse strategies for generating natural-language text

Artificial Intelligence
A statistical approach to machine translation

Computational Linguistics
The symbol grounding problem

CNLS '89 Proceedings of the ninth annual international conference of the Center for Nonlinear Studies on Self-organizing, Collective, and Cooperative Phenomena in Natural and Artificial Computing Networks on Emergent computation
Visual semantics: extracting visual information from text accompanying pictures

AAAI '94 Proceedings of the twelfth national conference on Artificial intelligence (vol. 1)
An efficient probabilistic context-free parsing algorithm that computes prefix probabilities

Computational Linguistics
Inductive Logic Programming: Techniques and Applications

Inductive Logic Programming: Techniques and Applications
The theory of parsing, translation, and compiling

The theory of parsing, translation, and compiling
Text Categorization with Suport Vector Machines: Learning with Many Relevant Features

ECML '98 Proceedings of the 10th European Conference on Machine Learning
A systematic comparison of various statistical alignment models

Computational Linguistics
Text classification using string kernels

The Journal of Machine Learning Research
Kernel methods for relation extraction

The Journal of Machine Learning Research
Matching words and pictures

The Journal of Machine Learning Research
The mathematics of statistical machine translation: parameter estimation

Computational Linguistics - Special issue on using large corpora: II
Building a large annotated corpus of English: the penn treebank

Computational Linguistics - Special issue on using large corpora: II
Introduction to the special issue on word sense disambiguation: the state of the art

Computational Linguistics - Special issue on word sense disambiguation
Kernel Methods for Pattern Analysis

Kernel Methods for Pattern Analysis
Two-level, many-paths generation

ACL '95 Proceedings of the 33rd annual meeting on Association for Computational Linguistics
A uniform architecture for parsing and generation

COLING '88 Proceedings of the 12th conference on Computational linguistics - Volume 2
A syntax-based statistical translation model

ACL '01 Proceedings of the 39th Annual Meeting on Association for Computational Linguistics
New ranking algorithms for parsing and tagging: kernels over discrete structures, and the voted perceptron

ACL '02 Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
BLEU: a method for automatic evaluation of machine translation

ACL '02 Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Lexicalized stochastic modeling of constraint-based grammars using log-linear measures and EM training

ACL '00 Proceedings of the 38th Annual Meeting on Association for Computational Linguistics
Bootstrapping lexical choice via multiple-sequence alignment

EMNLP '02 Proceedings of the ACL-02 conference on Empirical methods in natural language processing - Volume 10
Statistical acquisition of content selection rules for natural language generation

EMNLP '03 Proceedings of the 2003 conference on Empirical methods in natural language processing
A hierarchical phrase-based model for statistical machine translation

ACL '05 Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics
Using string-kernels for learning semantic parsers

ACL-44 Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics
Learning what to talk about in descriptive games

HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing
Collective content selection for concept-to-text generation

HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing
Learning for semantic parsing with statistical machine translation

HLT-NAACL '06 Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics
Automatic evaluation of machine translation quality using n-gram co-occurrence statistics

HLT '02 Proceedings of the second international conference on Human Language Technology Research
Learning to sportscast: a test of grounded language acquisition

Proceedings of the 25th international conference on Machine learning
On the integration of grounding language and learning objects

AAAI'04 Proceedings of the 19th national conference on Artifical intelligence
A generative model for parsing natural language to meaning representations

EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Situated models of meaning for sports video retrieval

NAACL-Short '07 Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers
A robot that uses existing vocabulary to infer non-visual word meanings from observation

AAAI'07 Proceedings of the 22nd national conference on Artificial intelligence - Volume 1
Learning language semantics from ambiguous supervision

AAAI'07 Proceedings of the 22nd national conference on Artificial intelligence - Volume 1
Name-it: naming and detecting faces in video by the integration of image and natural language processing

IJCAI'97 Proceedings of the Fifteenth international joint conference on Artifical intelligence - Volume 2
Database-text alignment via structured multilabel classification

IJCAI'07 Proceedings of the 20th international joint conference on Artifical intelligence
Interpreting written how-to instructions

IJCAI'09 Proceedings of the 21st international jont conference on Artifical intelligence
Reinforcement learning for mapping instructions to actions

ACL '09 Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - Volume 1
Learning semantic correspondences with less supervision

ACL '09 Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - Volume 1
Learning a compositional semantic parser using an existing syntactic parser

ACL '09 Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2
A statistical semantic parser that integrates syntax and semantics

CONLL '05 Proceedings of the Ninth Conference on Computational Natural Language Learning

Generative alignment and semantic parsing for learning from ambiguous supervision

COLING '10 Proceedings of the 23rd International Conference on Computational Linguistics: Posters
Learning language from its perceptual context

PADL'11 Proceedings of the 13th international conference on Practical aspects of declarative languages
Extracting aspects of determiner meaning from dialogue in a virtual world environment

IWCS '11 Proceedings of the Ninth International Conference on Computational Semantics
BALLGAME: a corpus for computational semantics

IWCS '11 Proceedings of the Ninth International Conference on Computational Semantics
Bootstrapping semantic parsers from conversations

EMNLP '11 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Reducing grounded learning tasks to grammatical inference

EMNLP '11 Proceedings of the Conference on Empirical Methods in Natural Language Processing
A semi supervised learning model for mapping sentences to logical form with ambiguous supervision

NLDB'12 Proceedings of the 17th international conference on Applications of Natural Language Processing and Information Systems
A probabilistic model of syntactic and semantic acquisition from child-directed utterances and their meanings

EACL '12 Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics
Unsupervised PCFG induction for grounded language learning with highly ambiguous supervision

EMNLP-CoNLL '12 Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning

Quantified Score

Hi-index	0.00

Visualization

Abstract

We present a novel framework for learning to interpret and generate language using only perceptual context as supervision. We demonstrate its capabilities by developing a system that learns to sportscast simulated robot soccer games in both English and Korean without any language-specific prior knowledge. Training employs only ambiguous supervision consisting of a stream of descriptive textual comments and a sequence of events extracted from the simulation trace. The system simultaneously establishes correspondences between individual comments and the events that they describe while building a translation model that supports both parsing and generation. We also present a novel algorithm for learning which events are worth describing. Human evaluations of the generated commentaries indicate they are of reasonable quality and in some cases even on par with those produced by humans for our limited domain.