Evaluating discourse understanding in spoken dialogue systems

Authors:
Ryuichiro Higashinaka;Noboru Miyazaki;Mikio Nakano;Kiyoaki Aikawa
Affiliations:
NTT Communication Science Laboratories, NTT Corporation, Tokyo, Japan;NTT Communication Science Laboratories, NTT Corporation, Tokyo, Japan;NTT Communication Science Laboratories, NTT Corporation, Tokyo, Japan;NTT Communication Science Laboratories, NTT Corporation, Tokyo, Japan
Venue:
ACM Transactions on Speech and Language Processing (TSLP)
Year:
2004

Citing 19
Cited 3

An object-oriented approach to dialogue management in spoken language systems

CHI '94 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
The nature of statistical learning theory

The nature of statistical learning theory
How may I help you?

Speech Communication - Special issue on interactive voice technology for telecommunication applications (IVITA '96)
Data mining: practical machine learning tools and techniques with Java implementations

Data mining: practical machine learning tools and techniques with Java implementations
The LIMSI ARISE system

Speech Communication - Special issue on interactive voice technology for telecommunication applications
An architecture for more realistic conversational systems

Proceedings of the 6th international conference on Intelligent user interfaces
Spoken dialogue technology: enabling the conversational user interface

ACM Computing Surveys (CSUR)
Plan Recognition in Natural Language Dialogue

Plan Recognition in Natural Language Dialogue
Collagen: applying collaborative discourse theory to human-computer interaction

AI Magazine
Vector-based natural language call routing

Computational Linguistics
Towards developing general models of usability with PARADISE

Natural Language Engineering
MIMIC: an adaptive mixed initiative spoken dialogue system for information queries

ANLC '00 Proceedings of the sixth conference on Applied natural language processing
PARADISE: a framework for evaluating spoken dialogue agents

ACL '98 Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
Flexible mixed-initiative dialogue management using concept-level confidence measures of speech recognizer output

COLING '00 Proceedings of the 18th conference on Computational linguistics - Volume 1
Construct Algebra: analytical dialog management

ACL '99 Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics
Extracting important sentences with support vector machines

COLING '02 Proceedings of the 19th international conference on Computational linguistics - Volume 1
Corpus-based discourse understanding in spoken dialogue systems

ACL '03 Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1
WIT: a toolkit for building robust and real-time spoken dialogue systems

SIGDIAL '00 Proceedings of the 1st SIGdial workshop on Discourse and dialogue - Volume 10
Comparing several aspects of human-computer and human-human dialogues

SIGDIAL '01 Proceedings of the Second SIGdial Workshop on Discourse and Dialogue - Volume 16

Stochastic discourse modeling in spoken dialogue systems using semantic dependency graphs

COLING-ACL '06 Proceedings of the COLING/ACL on Main conference poster sessions
A spoken dialogue system based on keyword spotting technology

HCI'07 Proceedings of the 12th international conference on Human-computer interaction: intelligent multimodal interaction environments
Issues in predicting user satisfaction transitions in dialogues: individual differences, evaluation criteria, and prediction models

IWSDS'10 Proceedings of the Second international conference on Spoken dialogue systems for ambient environments

Quantified Score

Hi-index	0.00

Visualization

Abstract

This article describes a method for creating an evaluation measure for discourse understanding in spoken dialogue systems. No well-established measure has yet been proposed for evaluating discourse understanding, which has made it necessary to evaluate it only on the basis of the system's total performance. Such evaluations, however, are greatly influenced by task domains and dialogue strategies. To find a measure that enables good estimation of system performance only from discourse understanding results, we enumerated possible discourse-understanding-related metrics and calculated their correlation with the system's total performance through dialogue experiments.