Beyond class A: a proposal for automatic evaluation of discourse
HLT '90 Proceedings of the workshop on Speech and Natural Language
Statistical methods for speech recognition
Statistical methods for speech recognition
Inductive learning algorithms and representations for text categorization
Proceedings of the seventh international conference on Information and knowledge management
Dialog Parsing in the TRAINS System
Dialog Parsing in the TRAINS System
Learning optimal dialogue strategies: a case study of a spoken dialogue agent for email
COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 2
Automatic optimization of dialogue management
COLING '00 Proceedings of the 18th conference on Computational linguistics - Volume 1
Resolving pronominal reference to abstract entities
ACL '02 Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Spoken dialogue management using probabilistic reasoning
ACL '00 Proceedings of the 38th Annual Meeting on Association for Computational Linguistics
Experiments in evaluating interactive spoken language systems
HLT '91 Proceedings of the workshop on Speech and Natural Language
Learning more effective dialogue strategies using limited dialogue move features
ACL-44 Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics
Comparing the utility of state features in spoken dialogue using reinforcement learning
HLT-NAACL '06 Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics
Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning
HLT '02 Proceedings of the second international conference on Human Language Technology Research
ITSPOKE: an intelligent tutoring spoken dialogue system
HLT-NAACL--Demonstrations '04 Demonstration Papers at HLT-NAACL 2004
Optimizing dialogue management with reinforcement learning: experiments with the NJFun system
Journal of Artificial Intelligence Research
Active learning in partially observable markov decision processes
ECML'05 Proceedings of the 16th European conference on Machine Learning
Inferring tutorial dialogue structure with hidden Markov modeling
EdAppsNLP '09 Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications
Discovering Tutorial Dialogue Strategies with Hidden Markov Models
Proceedings of the 2009 conference on Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling
To Elicit Or To Tell: Does It Matter?
Proceedings of the 2009 conference on Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling
Setting up user action probabilities in user simulations for dialog system development
ACL '09 Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2
Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system
Computer Speech and Language
Leveraging hidden dialogue state to select tutorial moves
IUNLPBEA '10 Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications
Spoken language understanding via supervised learning and linguistically motivated features
NLDB'10 Proceedings of the Natural language processing and information systems, and 15th international conference on Applications of natural language to information systems
User Modeling and User-Adapted Interaction
Automated decision making based on weak orderings
International Journal of Intelligent Information and Database Systems
Inducing effective pedagogical strategies using learning context features
UMAP'10 Proceedings of the 18th international conference on User Modeling, Adaptation, and Personalization
Characterizing the effectiveness of tutorial dialogue with hidden markov models
ITS'10 Proceedings of the 10th international conference on Intelligent Tutoring Systems - Volume Part I
ITS'10 Proceedings of the 10th international conference on Intelligent Tutoring Systems - Volume Part I
International Journal of Artificial Intelligence in Education - Special issue on Best of ITS 2010
International Journal of Artificial Intelligence in Education - Special issue on Best of ITS 2010
Reinforcement learning of question-answering dialogue policies for virtual museum guides
SIGDIAL '12 Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue
| Hi-index | 0.00 |
Although dialogue systems have been an area of research for decades, finding accurate ways of evaluating different systems is still a very active subfield since many leading methods, such as task completion rate or user satisfaction, capture different aspects of the end-to-end human-computer dialogue interaction. In this work, we step back the focus from the complete evaluation of a dialogue system to presenting metrics for evaluating one internal component of a dialogue system: its dialogue manager. Specifically, we investigate how to create and evaluate the best state space representations for a Reinforcement Learning model to learn an optimal dialogue control strategy. We present three metrics for evaluating the impact of different state models and demonstrate their use on the domain of a spoken dialogue tutoring system by comparing the relative utility of adding three features to a model of user, or student, state. The motivation for this work is that if one knows which features are best to use, one can construct a better dialogue manager, and thus better performing dialogue systems.