A study toward an evaluation method for spoken dialogue systems considering user criteria

Authors:
Etsuo Mizukami;Hideki Kashioka;Hisashi Kawai;Satoshi Nakamura
Affiliations:
National Institute of Information and Communications Technology, Kyoto, Japan;National Institute of Information and Communications Technology, Kyoto, Japan;National Institute of Information and Communications Technology, Kyoto, Japan;National Institute of Information and Communications Technology, Kyoto, Japan
Venue:
IWSDS'10 Proceedings of the Second international conference on Spoken dialogue systems for ambient environments
Year:
2010

Citing 2
Cited 1

PARADISE: a framework for evaluating spoken dialogue agents

ACL '98 Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
Statistical dialog management applied to WFST-based dialog systems

ICASSP '09 Proceedings of the 2009 IEEE International Conference on Acoustics, Speech and Signal Processing

Evaluation of facial direction estimation from cameras for multi-modal spoken dialog system

IWSDS'10 Proceedings of the Second international conference on Spoken dialogue systems for ambient environments

Quantified Score

Hi-index	0.00

Visualization

Abstract

In the development cycle of a spoken dialogue system (SDS), it is important to know how users actually behave and talk and what they expect of the SDS. We are developing SDSs which realize natural communication between users and systems. To collect users' real data, a wide-scale experiment was carried out with a smart-phone prototype SDS. In this brief paper, we report on the experiment's results and make a tentative analysis of cases in which there were gaps between system performance and user judgment. This requires both an adequate experimental design and an evaluation methodology that considers users' judgement criteria.