Evaluating interactive dialogue systems: extending component evaluation to integrated system evaluation

Authors:
Marilyn A. Walker;Diane J. Litman;Candace A. Kamm;Alicia Abella
Affiliations:
AT&T Labs--Research, Florham Park, NJ;AT&T Labs--Research, Florham Park, NJ;AT&T Labs--Research, Florham Park, NJ;AT&T Labs-Research, Florham Park, NJ
Venue:
ISDS '97 Interactive Spoken Dialog Systems on Bringing Speech and NLP Together in Real Applications
Year:
1997

Citing 17
Cited 3

Attention, intentions, and the structure of discourse

Computational Linguistics
A proposal for incremental dialogue evaluation

HLT '91 Proceedings of the workshop on Speech and Natural Language
Beyond class A: a proposal for automatic evaluation of discourse

HLT '90 Proceedings of the workshop on Speech and Natural Language
User Interfaces for voice applications

Voice communication between humans and machines
Spoken natural language dialog systems: a practical approach

Spoken natural language dialog systems: a practical approach
Empirical methods for artificial intelligence

Empirical methods for artificial intelligence
The effect of resource limits and task complexity on collaborative planning in dialogue

Artificial Intelligence - Special volume on empirical methods
Development Principles for Dialog-Based Interfaces

ECAI '96 Workshop on Dialogue Processing in Spoken Language Systems
Discourse segmentation by human and automated means

Computational Linguistics
Effects of variable initiative on linguistic behavior in human-computer spoken natural language dialogue

Computational Linguistics
PARADISE: a framework for evaluating spoken dialogue agents

ACL '98 Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
Evaluating discourse processing algorithms

ACL '89 Proceedings of the 27th annual meeting on Association for Computational Linguistics
A prosodic analysis of discourse segments in direction-giving monologues

ACL '96 Proceedings of the 34th annual meeting on Association for Computational Linguistics
Experiments in evaluating interactive spoken language systems

HLT '91 Proceedings of the workshop on Speech and Natural Language
Subject-based evaluation measures for interactive spoken language systems

HLT '91 Proceedings of the workshop on Speech and Natural Language
Human-machine problem solving using spoken language systems (SLS): factors affecting performance and user satisfaction

HLT '91 Proceedings of the workshop on Speech and Natural Language
Multi-site data collection and evaluation in spoken language understanding

HLT '93 Proceedings of the workshop on Human Language Technology

Usability issues in spoken dialogue systems

Natural Language Engineering
Usability evaluation in spoken language dialogue systems

ELDS '01 Proceedings of the workshop on Evaluation for Language and Dialogue Systems - Volume 9
Reinforcement learning for parameter estimation in statistical spoken dialogue systems

Computer Speech and Language

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper discusses the range of ways in which spoken dialogue system components have been evaluated and discusses approaches to evaluation that attempt to integrate component evaluation into an overall view of system performance. We will argue that the PARADISE (PARAdigm for DIalogue System Evaluation) framework has several advantages over other proposals.