Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation
Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation
International standards for HCI and usability
International Journal of Human-Computer Studies
PARADISE: a framework for evaluating spoken dialogue agents
ACL '98 Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
A meeting browser evaluation test
CHI '05 Extended Abstracts on Human Factors in Computing Systems
The AMI meeting corpus: a pre-announcement
MLMI'05 Proceedings of the Second international conference on Machine Learning for Multimodal Interaction
Shallow dialogue processing using machine learning algorithms (or not)
MLMI'04 Proceedings of the First international conference on Machine Learning for Multimodal Interaction
| Hi-index | 0.00 |
This paper outlines first the BET method for task-based evaluation of meeting browsers. 'Observations of interest' in meetings are empirically determined by neutral observers and then processed and ordered by evaluators. The evaluation of the TQB annotation-driven meeting browser using the BET is then described. A series of subjects attempted to answer as many meeting-related questions as possible in a fixed amount of time, and their performance was measured in terms of precision and speed. The results indicate that the TQB interface is easy to understand with little prior learning and that its annotation-based search functionality is highly relevant, in particular keyword search over the meeting transcript. Two knowledge-poorer browsers appear to offer lower precision but higher speed. The BET task-based evaluation method thus appears to be a coherent measure of browser quality.