Semi-formal Evaluation of Conversational Characters

Authors:
Ron Artstein;Sudeep Gandhe;Jillian Gerten;Anton Leuski;David Traum
Affiliations:
Institute for Creative Technologies, University of Southern California, Marina del Rey, USA CA 90292;Institute for Creative Technologies, University of Southern California, Marina del Rey, USA CA 90292;Institute for Creative Technologies, University of Southern California, Marina del Rey, USA CA 90292;Institute for Creative Technologies, University of Southern California, Marina del Rey, USA CA 90292;Institute for Creative Technologies, University of Southern California, Marina del Rey, USA CA 90292
Venue:
Languages: From Formal to Natural
Year:
2009

Citing 5
Cited 4

Building a large annotated corpus of English: the penn treebank

Computational Linguistics - Special issue on using large corpora: II
Inter-coder agreement for computational linguistics

Computational Linguistics
An application of reinforcement learning to dialogue strategy selection in a spoken dialogue system for email

Journal of Artificial Intelligence Research
Building effective question answering characters

SigDIAL '06 Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue
Dealing with out of domain questions in virtual characters

IVA'06 Proceedings of the 6th international conference on Intelligent Virtual Agents

Ada and grace: toward realistic and engaging virtual museum guides

IVA'10 Proceedings of the 10th international conference on Intelligent virtual agents
I've said it before, and I'll say it again: an empirical investigation of the upper bound of the selection approach to dialogue

SIGDIAL '10 Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue
Error return plots

SIGDIAL '11 Proceedings of the SIGDIAL 2011 Conference
The differences in motivations of online game players and offline game players: A combined analysis of three studies at higher education level

Computers & Education

Quantified Score

Hi-index	0.00

Visualization

Abstract

Conversational dialogue systems cannot be evaluated in a fully formal manner, because dialogue is heavily dependent on context and current dialogue theory is not precise enough to specify a target output ahead of time. Instead, we evaluate dialogue systems in a semi-formal manner, using human judges to rate the coherence of a conversational character and correlating these judgments with measures extracted from within the system. We present a series of three evaluations of a single conversational character over the course of a year, demonstrating how this kind of evaluation helps bring about an improvement in overall dialogue coherence.