Fundamentals of speech recognition
Fundamentals of speech recognition
Natural language understanding (2nd ed.)
Natural language understanding (2nd ed.)
The LIMSI RailTel system: field trial of a telephone service for rail travel information
Speech Communication - Special issue on interactive voice technology for telecommunication applications (IVITA '96)
PADIS—an automatic telephone switchboard and directory information system
Speech Communication - Special issue on interactive voice technology for telecommunication applications (IVITA '96)
Speech Communication - Special issue on interactive voice technology for telecommunication applications (IVITA '96)
Automatic Evaluation Environment for Spoken Dialogue Systems
ECAI '96 Workshop on Dialogue Processing in Spoken Language Systems
Dialogos: A Robust System for Human-Machine Spoken Dialogue on the Telephone
ICASSP '97 Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '97)-Volume 2 - Volume 2
Broadcast News Transcription Using HTK
ICASSP '97 Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '97)-Volume 2 - Volume 2
The 1998 HTK system for transcription of conversational telephone speech
ICASSP '99 Proceedings of the Acoustics, Speech, and Signal Processing, 1999. on 1999 IEEE International Conference - Volume 01
The Knowledge Engineering Review
Developing a flexible spoken dialog system using simulation
ACL '04 Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics
Testing the performance of spoken dialogue systems by means of an artificially simulated user
Artificial Intelligence Review
User simulation in a stochastic dialog system
Computer Speech and Language
Example-based dialog modeling for practical multi-domain dialog system
Speech Communication
Data-driven user simulation for automated evaluation of spoken dialog systems
Computer Speech and Language
Analysis of a new simulation approach to dialog system evaluation
Speech Communication
Comparing user simulation models for dialog strategy learning
NAACL-Short '07 Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers
User simulation as testing for spoken dialog systems
SIGdial '08 Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue
A framework for model-based evaluation of spoken dialog systems
SIGdial '08 Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue
Setting up user action probabilities in user simulations for dialog system development
ACL '09 Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2
Modeling user satisfaction with Hidden Markov Model
SIGDIAL '09 Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Using knowledge of misunderstandings to increase the robustness of spoken dialogue systems
Knowledge-Based Systems
Semi-automatic creation of resources for spoken dialog systems
KI'09 Proceedings of the 32nd annual German conference on Advances in artificial intelligence
Hybrid user intention modeling to diversify dialog simulations
Computer Speech and Language
Using knowledge about misunderstandings to increase the robustness of spoken dialogue systems
TSD'10 Proceedings of the 13th international conference on Text, speech and dialogue
IWSDS'10 Proceedings of the Second international conference on Spoken dialogue systems for ambient environments
Introduction to special issue on machine learning for adaptivity in spoken dialogue systems
ACM Transactions on Speech and Language Processing (TSLP)
Human-machine corpus analysis for generation and interaction with spoken dialog systems
KI'11 Proceedings of the 34th Annual German conference on Advances in artificial intelligence
| Hi-index | 0.00 |
In recent years, a question of great interest has been the development of tools and techniqnes to facilitate the evaluation of dialogue systems. The latter can be evaluated from various points of view, such as recognition and understanding rates, dialogue naturalness and robustness against recognition errors. Evaluation usually requires compiling a large corpus of words and sentences uttered by users, relevant to the application domain the system is designed for. This paper proposes a new technique that makes it possible to reuse such a corpus for the evaluation and to check the performance of the system when different dialogue strategies are used. The technique is based on the automatic generation of conversations between the dialogue system, together with an additional dialogue system called user simulator that represents the user's interaction with the dialogue system. The technique has been applied to evaluate a dialogue system developed in our lab using two different recognition front-ends and two different dialogue strategies to handle user confirmations. The experiments show that the prompt-dependent recognition front-end achieves better results, but that this front-end is appropriate only if users limit their utterances to those related to the current system prompt. The prompt-independent front-end achieves inferior results, but enables front-end users to utter any permitted utterance at any time, irrespective of the system prompt. In consequence, this front-end may allow a more natural and comfortable interaction. The experiments also show that the re-prompting confirmation strategy enhances system performance for both recognition front-ends.