Something from nothing: augmenting a paper-based work practice via multimodal interaction

Authors:
David R. McGee;Philip R. Cohen;Lizhong Wu
Affiliations:
Center for Human Computer Communication, Department of Computer Science and Engineering, Oregon Graduate Institute, Portland, OR;Center for Human Computer Communication, Department of Computer Science and Engineering, Oregon Graduate Institute, Portland, OR;Center for Human Computer Communication, Department of Computer Science and Engineering, Oregon Graduate Institute, Portland, OR
Venue:
DARE '00 Proceedings of DARE 2000 on Designing augmented reality environments
Year:
2000

Citing 25
Cited 29

Intelligent multi-media interface technology

Intelligent user interfaces
The logic of typed feature structures

The logic of typed feature structures
A desk supporting computer-based interaction with paper documents

CHI '92 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Some computer science issues in ubiquitous computing

Communications of the ACM - Special issue on computer augmented environments: back to the real world
Interacting with paper on the DigitalDesk

Communications of the ACM - Special issue on computer augmented environments: back to the real world
Integrating simultaneous input from speech, gaze, and hand gestures

Intelligent multimedia interfaces
A generic platform for addressing the multimodal challenge

CHI '95 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Tangible bits: towards seamless interfaces between people, bits and atoms

Proceedings of the ACM SIGCHI Conference on Human factors in computing systems
Integration and synchronization of input modes during multimodal human-computer interaction

Proceedings of the ACM SIGCHI Conference on Human factors in computing systems
The metaDESK: models and prototypes for tangible user interfaces

Proceedings of the 10th annual ACM symposium on User interface software and technology
QuickSet: multimodal interaction for distributed applications

MULTIMEDIA '97 Proceedings of the fifth ACM international conference on Multimedia
mediaBlocks: physical containers, transports, and controls for online media

Proceedings of the 25th annual conference on Computer graphics and interactive techniques
An open agent architecture

Readings in agents
Design principles for intelligent environments

AAAI '98/IAAI '98 Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence
Bridging physical and virtual worlds with electronic tags

Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Urp: a luminous-tangible workbench for urban planning and design

Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Integrated interfaces for decision-support with simulation

WSC '91 Proceedings of the 23rd conference on Winter simulation
The ins and outs of collaborative walls: demonstrating the collaborage concept

CHI '99 Extended Abstracts on Human Factors in Computing Systems
The efficiency of multimodal interaction for a map-based task

CHI '00 Extended Abstracts on Human Factors in Computing Systems
“Put-that-there”: Voice and gesture at the graphics interface

SIGGRAPH '80 Proceedings of the 7th annual conference on Computer graphics and interactive techniques
Unification-based multimodal integration

ACL '98 Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
Confirmation in multimodal systems

COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 2
Unification-based multimodal parsing

COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 1
Multimodal interactive maps: designing for human performance

Human-Computer Interaction
Multimodal integration-a statistical view

IEEE Transactions on Multimedia

Creating tangible interfaces by augmenting physical objects with multimodal language

Proceedings of the 6th international conference on Intelligent user interfaces
The designers' outpost: a tangible interface for collaborative web site

Proceedings of the 14th annual ACM symposium on User interface software and technology
Comparing paper and tangible, multimodal tools

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
A framework to manage multimodal fusion of events for advanced interactions within virtual environments

EGVE '02 Proceedings of the workshop on Virtual environments 2002
The missing link: augmenting biology laboratory notebooks

Proceedings of the 15th annual ACM symposium on User interface software and technology
Designing a human-centered, multimodal GIS interface to support emergency management

Proceedings of the 10th ACM international symposium on Advances in geographic information systems
Two worlds apart: bridging the gap between physical and virtual media for distributed design collaboration

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Architectures multi-agents génériques à base de réseaux de Pétri colorés temporisés pour la fusion multimodale en entrée

IHM '02 Proceedings of the 14th French-speaking conference on Human-computer interaction (Conférence Francophone sur l'Interaction Homme-Machine)
Tangible multimodal interfaces for safety-critical applications

Communications of the ACM - Multimodal interfaces that flex, adapt, and persist
A visual modality for the augmentation of paper

Proceedings of the 2001 workshop on Perceptive user interfaces
Naturally conveyed explanations of device behavior

Proceedings of the 2001 workshop on Perceptive user interfaces
Only touching the surface: creating affinities between digital content and paper

CSCW '04 Proceedings of the 2004 ACM conference on Computer supported cooperative work
Localisation and Interaction for Augmented Maps

ISMAR '05 Proceedings of the 4th IEEE/ACM International Symposium on Mixed and Augmented Reality
Naturally conveyed explanations of device behavior

ACM SIGGRAPH 2006 Courses
Generic multimedia multimodal agents paradigms and their dynamic reconfiguration at the architectural level

EURASIP Journal on Applied Signal Processing
Designing the user interface for multimodal speech and pen-based gesture applications: state-of-the-art systems and future research directions

Human-Computer Interaction
Context shifts: extending the meanings of physical objects with language

Human-Computer Interaction
Implementation of god-like interaction techniques for supporting collaboration between outdoor AR and indoor tabletop users

ISMAR '06 Proceedings of the 5th IEEE and ACM International Symposium on Mixed and Augmented Reality
Like bees around the hive: a comparative study of a mobile augmented reality map

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
User-designed information tools to support communication and care coordination in a trauma hospital

Journal of Biomedical Informatics
Ubiquitous Computing for Capture and Access

Foundations and Trends in Human-Computer Interaction
Augmenting Sticky Notes as an I/O Interface

UAHCI '09 Proceedings of the 5th International on ConferenceUniversal Access in Human-Computer Interaction. Part II: Intelligent and Ubiquitous Interaction Environments
Is the Writing on the Wall for Tabletops?

INTERACT '09 Proceedings of the 12th IFIP TC 13 International Conference on Human-Computer Interaction: Part II
Multimodal interaction: a new focal area for AI

IJCAI'01 Proceedings of the 17th international joint conference on Artificial intelligence - Volume 2
Cartouche: conventions for tangibles bridging diverse interactive systems

Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction
Augmented paper: developing relationships between digital content and paper

The disappearing computer
An ego-centric and tangible approach to meeting indexing and browsing

MLMI'07 Proceedings of the 4th international conference on Machine learning for multimodal interaction
Putting location-based services on the map

W2GIS'06 Proceedings of the 6th international conference on Web and Wireless Geographical Information Systems
Shared design space: sketching ideas using digital pens and a large augmented tabletop setup

ICAT'06 Proceedings of the 16th international conference on Advances in Artificial Reality and Tele-Existence

Quantified Score

Hi-index	0.00

Visualization

Abstract

In this paper, we describe Rasa: an environment designed to augment, rather than replace, the work habits of its users. These work habits include drawing on Post-it™ notes using a symbolic language. Rasa observes and understands this language, assigning meaning simultaneously to objects in both the physical and virtual worlds. With Rasa, users rollout a paper map, register it, and move the augmented objects from one place to another on it. Once an object is augmented, users can modify the meaning represented by it, ask questions about that representation, view it in virtual reality, or give directions to it, all with speech and gestures. We examine the way Rasa uses language to augment objects, and compare it with prior methods, arguing that language is a more visible, flexible, and comprehensible method for creating augmentations than other approaches.