Augmenting environments with multimodal interaction

• Authors: Philip R. Cohen;David Ray Mcgee
• Venue: Augmenting environments with multimodal interaction
• Year: 2003

In air-traffic control centers, military command posts, and hospital emergency rooms, life-and-death decisions must be made quickly in the face of uncertain information. In these high-stress environments, professionals put a premium on safety, timeliness, team cohesiveness, and mutual awareness. Consequently, these professionals discard computational tools in favor of those that are more robust, malleable, physical, and high in resolution. Indeed, we claim that in order for computational tools to be effective and accepted, they must support these properties. Within this thesis, we argue that it is now possible, using the language of a task, to augment real world artifacts, creating digital tools as robust, malleable, and portable as paper and other physical artifacts. In support of this thesis, we developed Rasa, a tangible augmented reality environment that digitally enhances the existing paper-based command-and control-capability in a military command post. By observing and understanding the users' speech, pen, and touch-based multimodal language, Rasa computationally augments the physical objects on a command post map, linking these items to their digital representations—for example, linking a paper map to the world and Post-it™ notes to military units. Herein, we (1) argue that the properties of physical tools cannot be ignored when designing computational replacements, and demonstrate the effect of doing so in a military command post; (2) identify constraints for the design of fail-safe, mission-critical systems for decision support; (3) present Rasa, a system for augmenting physical objects, transforming them into tangible interfaces; (4) review the related work in augmented reality and paper-based and invisible interfaces; (5) present the findings of an experimental evaluation of Rasa in the field; (6) discuss the results of this evaluation, the limitations of Rasa, its relevance, and the overall impact of Rasa on the design of intelligent systems; (7) expound upon potential avenues of future work, including a vision for Rasa that includes the computational augmentation of arbitrary physical objects.