Optimizing dialogue management with reinforcement learning: experiments with the NJFun system

Authors:
Satinder Singh;Diane Litman;Michael Kearns;Marilyn Walker
Affiliations:
Syntek Capiral, New York, NY;Department of Computer Science and LRDC, University of Pittsburgh, Pittsburgh, PA;Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA;AT&T Labs-Research, Florham Park, NJ
Venue:
Journal of Artificial Intelligence Research
Year:
2002

Citing 14
Cited 59

User Interfaces for voice applications

Voice communication between humans and machines
Temporal difference learning and TD-Gammon

Communications of the ACM
An evaluation of strategies for selectively verifying utterance meanings in spoken natural language dialog

International Journal of Human-Computer Studies
Introduction to Reinforcement Learning

Introduction to Reinforcement Learning
Neuro-Dynamic Programming

Neuro-Dynamic Programming
Predicting and Adapting to Poor Speech Recognition in a Spoken Dialogue System

Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence
Tracking initiative in collaborative dialogue interactions

ACL '98 Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
Evaluating response strategies in a Web-based spoken dialogue agent

COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 2
Learning optimal dialogue strategies: a case study of a spoken dialogue agent for email

COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 2
Mixed initiative in dialogue: an investigation into discourse segmentation

ACL '90 Proceedings of the 28th annual meeting on Association for Computational Linguistics
Automatic detection of poor speech recognition at the dialogue level

ACL '99 Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics
SPoT: a trainable sentence planner

NAACL '01 Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies
An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email

Journal of Artificial Intelligence Research
Reinforcement learning: a survey

Journal of Artificial Intelligence Research

Query-by-critique: spoken language access to large lists

Proceedings of the 15th annual ACM symposium on User interface software and technology
Towards a theory of natural language interfaces to databases

Proceedings of the 8th international conference on Intelligent user interfaces
Adaptive cognitive orthotics: combining reinforcement learning and constraint-based temporal reasoning

ICML '04 Proceedings of the twenty-first international conference on Machine learning
A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies

The Knowledge Engineering Review
Learning more effective dialogue strategies using limited dialogue move features

ACL-44 Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics
Learning the structure of task-driven human-human dialogs

ACL-44 Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics
Error handling in the RavenClaw dialog management framework

HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing
Learning what to talk about in descriptive games

HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing
Learning mixed initiative dialog strategies by using reinforcement learning on both conversants

HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing
Partially observable Markov decision processes for spoken dialog systems

Computer Speech and Language
Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning

HLT '02 Proceedings of the second international conference on Human Language Technology Research
Testing the performance of spoken dialogue systems by means of an artificially simulated user

Artificial Intelligence Review
Compositional specification and realisation of mixed-initiative web dialogs

International Journal of Web Engineering and Technology
A Reinforcement Learning approach to evaluating state representations in spoken dialogue systems

Speech Communication
Automating spoken dialogue management design using machine learning: An industry perspective

Speech Communication
DialogStudio: A workbench for data-driven spoken dialog system development and management

Speech Communication
Evaluating user simulations with the Cramér-von Mises divergence

Speech Communication
Hybrid reinforcement/supervised learning of dialogue policies from fixed data sets

Computational Linguistics
Does this list contain what you were searching for? Learning adaptive dialogue strategies for interactive question answering

Natural Language Engineering
Learning effective and engaging strategies for advice-giving human-machine dialogue

Natural Language Engineering
Improving recommender systems with adaptive conversational strategies

Proceedings of the 20th ACM conference on Hypertext and hypermedia
Natural language generation as planning under uncertainty for spoken dialogue systems

EACL '09 Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics
Comparing user simulation models for dialog strategy learning

NAACL-Short '07 Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers
Evaluation of a hierarchical reinforcement learning spoken dialogue system

Computer Speech and Language
User simulation as testing for spoken dialog systems

SIGdial '08 Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue
A personalized system for conversational recommendations

Journal of Artificial Intelligence Research
Case-based techniques used for dialogue understanding and planning in a human-robot dialogue system

IJCAI'07 Proceedings of the 20th international joint conference on Artifical intelligence
Bayes risk-based dialogue management for document retrieval system with speech interface

Speech Communication
Using word-sense disambiguation methods to classify web queries by intent

EMNLP '09 Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 3 - Volume 3
Discourse structure and performance analysis: beyond the correlation

SIGDIAL '09 Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Review:

The Knowledge Engineering Review
Using knowledge of misunderstandings to increase the robustness of spoken dialogue systems

Knowledge-Based Systems
Semi-automatic task recognition for interactive narratives with EAT & RUN

Proceedings of the Intelligent Narrative Technologies III Workshop
Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system

Computer Speech and Language
Reading between the lines: learning to map high-level instructions to commands

ACL '10 Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics
Learning what to say and how to say it: Joint optimisation of spoken dialogue management and natural language generation

Computer Speech and Language
Natural language generation as planning under uncertainty for spoken dialogue systems

Empirical methods in natural language generation
ACE (Actor-Critic-Explorer) paradigm for reinforcement learning in basal ganglia: Highlighting the role of subthalamic and pallidal nuclei

Neurocomputing
SmartWeb handheld: multimodal interaction with ontological knowledge bases and semantic web services

ICMI'06/IJCAI'07 Proceedings of the ICMI 2006 and IJCAI 2007 international conference on Artifical intelligence for human computing
A multi-expert model for dialogue and behavior control of conversational robots and agents

Knowledge-Based Systems
Spatially-aware dialogue control using hierarchical reinforcement learning

ACM Transactions on Speech and Language Processing (TSLP)
Comparing user simulations for dialogue strategy learning

ACM Transactions on Speech and Language Processing (TSLP)
Empirically evaluating the application of reinforcement learning to the induction of effective and adaptive pedagogical strategies

User Modeling and User-Adapted Interaction
Learning and evaluation of dialogue strategies for new applications: Empirical methods for optimization from small data sets

Computational Linguistics
Intuition as instinctive dialogue

Computing with instinct
Semi-automated dialogue act classification for situated social agents in games

Agents for games and simulations II
Creating a general collaborative dialogue agent with lounge strategy feature

Expert Systems with Applications: An International Journal
Controlling user perceptions of linguistic style: Trainable generation of personality traits

Computational Linguistics
Fast reinforcement learning of dialogue policies using stable function approximation

IJCNLP'04 Proceedings of the First international joint conference on Natural Language Processing
Optimising natural language generation decision making for situated dialogue

SIGDIAL '11 Proceedings of the SIGDIAL 2011 Conference
Bootstrapping semantic parsers from conversations

EMNLP '11 Proceedings of the Conference on Empirical Methods in Natural Language Processing
Inducing effective pedagogical strategies using learning context features

UMAP'10 Proceedings of the 18th international conference on User Modeling, Adaptation, and Personalization
Building Intelligent Interactive Tutors: Student-centered strategies for revolutionizing e-learning

Building Intelligent Interactive Tutors: Student-centered strategies for revolutionizing e-learning
Learning automata-based approach to learn dialogue policies in large state space

International Journal of Intelligent Information and Database Systems
Bringing together commercial and academic perspectives for the development of intelligent AmI interfaces

Journal of Ambient Intelligence and Smart Environments - A software engineering perspective on smart applications for AmI
Optimising incremental dialogue decisions using information density for interactive systems

EMNLP-CoNLL '12 Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning
A learning approach to optimizing exploration---exploitation tradeoff in relevance feedback

Information Retrieval
A domain-independent statistical methodology for dialog management in spoken dialog systems

Computer Speech and Language
A dialogue management system using a corpus-based framework and a dynamic dialogue transition model

AI Communications

Quantified Score

Hi-index	0.00

Visualization

Abstract

Designing the dialogue policy of a spoken dialogue system involves many nontrivial choices. This paper presents a reinforcement learning approach for automatically optimizing a dialogue policy, which addresses the technical challenges in applying reinforcement learning to a working dialogue system with human users. We report on the design, construction and empirical evaluation of NJFun, an experimental spoken dialogue system that provides users with access to information about fun things to do in New Jersey. Our results show that by optimizing its performance via reinforcement learning, NJFun measurably improves system performance.