Automatic Evaluation Tool for Multimodal Dialogue Systems

Authors:
Hugo Wesseling;Matthias Bezold;Nicole Beringer
Affiliations:
Elektrobit Automotive GmbH, Erlangen, Germany 91058 and Institute of Information Technology, University of Ulm, Germany 89081;Elektrobit Automotive GmbH, Erlangen, Germany 91058 and Institute of Information Technology, University of Ulm, Germany 89081;Elektrobit Automotive GmbH, Erlangen, Germany 91058
Venue:
PIT '08 Proceedings of the 4th IEEE tutorial and research workshop on Perception and Interactive Technologies for Speech-Based Systems: Perception in Multimodal Dialogue Systems
Year:
2008

Citing 3
Cited 1

Towards developing general models of usability with PARADISE

Natural Language Engineering
Usability evaluation in spoken language dialogue systems

ELDS '01 Proceedings of the workshop on Evaluation for Language and Dialogue Systems - Volume 9
Model based HMI specification in an automotive context

Proceedings of the 2007 conference on Human interface: Part I

Describing User Interactions in Adaptive Interactive Systems

UMAP '09 Proceedings of the 17th International Conference on User Modeling, Adaptation, and Personalization: formerly UM and AH

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper presents a system built for improving the evaluation process of multimodal dialogue systems by providing a graphical representation of event-based recording data. Low-level information from recordings can visually be combined to form higher-level actions using pattern definitions to create a hierarchy of actions. Actions are visually represented by a multitrack player-like view where all modalities of the recording can be watched and manipulated. But besides tagging and derivation of higher-level actions, further help is provided by a correlation analysis and an export of charts and tables to Microsoft Excel. The tool is tested on recording data and questionnaire answers obtained from Wizard of Oz (WOZ) experiments performed within the DICIT project.