Multi-site data collection and evaluation in spoken language understanding

Authors:
L. Hirschman;M. Bates;D. Dahl;W. Fisher;J. Garofolo;D. Pallett;K. Hunicke-Smith;P. Price;A. Rudnicky;E. Tzoukermann
Affiliations:
MIT Laboratory for Computer Science, Cambridge, MA;MIT Laboratory for Computer Science, Cambridge, MA;MIT Laboratory for Computer Science, Cambridge, MA;MIT Laboratory for Computer Science, Cambridge, MA;MIT Laboratory for Computer Science, Cambridge, MA;MIT Laboratory for Computer Science, Cambridge, MA;MIT Laboratory for Computer Science, Cambridge, MA;MIT Laboratory for Computer Science, Cambridge, MA;MIT Laboratory for Computer Science, Cambridge, MA;MIT Laboratory for Computer Science, Cambridge, MA
Venue:
HLT '93 Proceedings of the workshop on Human Language Technology
Year:
1993

Citing 6
Cited 12

Procedure for quantitatively comparing the syntactic coverage of English grammars

HLT '91 Proceedings of the workshop on Speech and Natural Language
Evaluation of spoken language systems: the ATIS domain

HLT '90 Proceedings of the workshop on Speech and Natural Language
Developing an evaluation methodology for spoken language systems

HLT '90 Proceedings of the workshop on Speech and Natural Language
Multi-site data collection for a spoken language corpus

HLT '91 Proceedings of the workshop on Speech and Natural Language
Experiments in evaluating interactive spoken language systems

HLT '91 Proceedings of the workshop on Speech and Natural Language
Benchmark tests for the DARPA Spoken Language Program

HLT '93 Proceedings of the workshop on Human Language Technology

Head-Transducer Models for Speech Translation and TheirAutomatic Acquisition from Bilingual Data

Machine Translation
Empirical studies in discourse

Computational Linguistics
Andrew Radford. Syntactic Theory and the Structure of English: A minimalist approach. Cambridge University Press, 1997. £18.95, ISBN 0-521-47707-7.

Natural Language Engineering
A comparison of head transducers and transfer for a limited domain translation application

ACL '98 Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
Head automata and bilingual tiling: translation with minimal representations

ACL '96 Proceedings of the 34th annual meeting on Association for Computational Linguistics
Benchmark tests for the DARPA Spoken Language Program

HLT '93 Proceedings of the workshop on Human Language Technology
Expanding the scope of the ATIS task: the ATIS-3 corpus

HLT '94 Proceedings of the workshop on Human Language Technology
Session summary

HLT '94 Proceedings of the workshop on Human Language Technology
Experiences of an in-service Wizard-of-Oz data collection for the deployment of a call-routing application

NAACL-HLT-Dialog '07 Proceedings of the Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies
Evaluating interactive dialogue systems: extending component evaluation to integrated system evaluation

ISDS '97 Interactive Spoken Dialog Systems on Bringing Speech and NLP Together in Real Applications
Evaluation of a domain independent approach to natural language processing for game-like user interfaces

CIG'09 Proceedings of the 5th international conference on Computational Intelligence and Games
Evaluating semantic evaluations: how RTE measures up

MLCW'05 Proceedings of the First international conference on Machine Learning Challenges: evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment

Quantified Score

Hi-index	0.01

Visualization

Abstract

The Air Travel Information System (ATIS) domain serves as the common task for DARPA spoken language system research and development. The approaches and results possible in this rapidly growing area are structured by available corpora, annotations of that data, and evaluation methods. Coordination of this crucial infrastructure is the charter of the Multi-Site ATIS Data COllection Working group (MADCOW). We focus here on selection of training and test data, evaluation of language understanding, and the continuing search for evaluation methods that will correlate well with expected performance of the technology in applications.