Mixed-initiative dialog management for speech-based interaction with graphical user interfaces

• Authors: Andreas Löhr;Bernd Brügge
• Venue: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
• Year: 2008

Controlling graphical user interfaces (GUI) by speech is slow, but proves useful for disabled persons with limitations in operating mouse and keyboard. We present conversation-and-control, a new approach for using speech as input modality for GUIs, which facilitates direct manipulation of widget functions by spoken commands. Our approach is based on a command language, which provides a unique command for each specific widget function. For managing the interaction we propose a mixed-initiative dialog model, which can be generated from widget properties. Using heuristics for inferring the meaning of a recognition result and having the ability to ask clarification questions, our approach avoids the rejection of recognition errors. We hypothesized that conversation-and-control allows for shorter task completion times than conventional command-and-control approaches, due to a reduction of the average number of required commands. The results of a user experiment, which we present and discuss, indicate a 16.8% reduction of task completion time achieved by our approach.