Unified theories of cognition
The Psychology of Human-Computer Interaction
The Psychology of Human-Computer Interaction
Automatic Evaluation Environment for Spoken Dialogue Systems
ECAI '96 Workshop on Dialogue Processing in Spoken Language Systems
Assessment of dialogue systems by means of a new simulation technique
Speech Communication
Towards developing general models of usability with PARADISE
Natural Language Engineering
PARADISE: a framework for evaluating spoken dialogue agents
ACL '98 Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
Multipurpose Prototypes for Assessing User Interfaces in Pervasive Computing Systems
IEEE Pervasive Computing
Developing a flexible spoken dialog system using simulation
ACL '04 Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics
Agenda-based user simulation for bootstrapping a POMDP dialogue system
NAACL-Short '07 Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers
User simulation as testing for spoken dialog systems
SIGdial '08 Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue
Users interact differently: towards a usability-oriented user taxonomy
HCI'07 Proceedings of the 12th international conference on Human-computer interaction: interaction design and usability
A user model to predict user satisfaction with spoken dialog systems
IWSDS'10 Proceedings of the Second international conference on Spoken dialogue systems for ambient environments
Assessing user simulation for dialog systems using human judges and automatic evaluation measures
Natural Language Engineering
Evaluating user interface adaptations at runtime by simulating user interaction
BCS-HCI '11 Proceedings of the 25th BCS Conference on Human-Computer Interaction
International Journal of Speech Technology
Hi-index | 0.00 |
The evaluation of spoken dialog systems still relies on subjective interaction experiments for quantifying interaction behavior and user-perceived quality. In this paper, we present a simulation approach replacing subjective tests in early system design and evaluation phases. The simulation is based on a model of the system, and a probabilistic model of user behavior. Probabilities for the next user action vary in dependence of system features and user characteristics, as defined by rules. This way, simulations can be conducted before data have been acquired. In order to evaluate the simulation approach, characteristics of simulated interactions are compared to interaction corpora obtained in subjective experiments. As was previously proposed in the literature, we compare interaction parameters for both corpora and calculate recall and precision of user utterances. The results are compared to those from a comparison of real user corpora. While the real corpora are not equal, they are more similar than the simulation is to the real data. However, the simulations can predict differences between system versions and user groups quite well on a relative level. In order to derive further requirements for the model, we conclude with a detailed analysis of utterances missing in the simulated corpus and consider the believability of entire dialogs.