Finding errors automatically in semantically tagged dialogues

Authors:
John Aberdeen;Christine Doran;Laurie Damianos;Samuel Bayer;Lynette Hirschman
Affiliations:
The MITRE Corporation, Bedford, MA;The MITRE Corporation, Bedford, MA;The MITRE Corporation, Bedford, MA;The MITRE Corporation, Bedford, MA;The MITRE Corporation, Bedford, MA
Venue:
HLT '01 Proceedings of the first international conference on Human language technology research
Year:
2001

Citing 2
Cited 2

Mixed-initiative development of language processing systems

ANLC '97 Proceedings of the fifth conference on Applied natural language processing
Characterizing and recognizing spoken corrections in human-computer dialogue

COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 1

What's the trouble: automatically identifying problematic dialogues in DARPA communicator dialogue systems

ACL '02 Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Comparing several aspects of human-computer and human-human dialogues

SIGDIAL '01 Proceedings of the Second SIGdial Workshop on Discourse and Dialogue - Volume 16

Quantified Score

Hi-index	0.00

Visualization

Abstract

We describe a novel method for detecting errors in task-based human-computer (HC) dialogues by automatically deriving them from semantic tags. We examined 27 HC dialogues from the DARPA Communicator air travel domain, comparing user inputs to system responses to look for slot value discrepancies, both automatically and manually. For the automatic method, we labeled the dialogues with semantic tags corresponding to "slots" that would be filled in "frames" in the course of the travel task. We then applied an automatic algorithm to detect errors in the dialogues. The same dialogues were also manually tagged (by a different annotator) to label errors directly. An analysis of the results of the two tagging methods indicates that it may be possible to detect errors automatically in this way, but our method needs further work to reduce the number of false errors detected. Finally, we present a discussion of the differing results from the two tagging methods.