Speech and sketching for multimodal design

Authors:
Aaron Adler;Randall Davis
Affiliations:
MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA;MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA
Venue:
Proceedings of the 9th international conference on Intelligent user interfaces
Year:
2004

Citing 4
Cited 15

Towards a computational model of sketching

Proceedings of the 6th international conference on Intelligent user interfaces
MATCH: an architecture for multimodal dialogue systems

ACL '02 Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Designing the user interface for multimodal speech and pen-based gesture applications: state-of-the-art systems and future research directions

Human-Computer Interaction
Resolving ambiguities to create a natural computer-based sketching environment

IJCAI'01 Proceedings of the 17th international joint conference on Artificial intelligence - Volume 2

Speech, ink, and slides: the interaction of content channels

Proceedings of the 12th annual ACM international conference on Multimedia
Voice recognition technology for visual artists with disabilities in their upper limbs

OZCHI '05 Proceedings of the 17th Australia conference on Computer-Human Interaction: Citizens Online: Considerations for Today and the Future
How smart are our environments? An updated look at the state of the art

Pervasive and Mobile Computing
The drive to create: an investigation of tools to support disabled artists

Proceedings of the 6th ACM SIGCHI conference on Creativity & cognition
Sketch understanding systems

ACM SIGGRAPH 2007 courses
Bibliography

ACM SIGGRAPH 2007 courses
Magic Paper: Sketch-Understanding Research

Computer
The effect of representation of triggers on design outcomes

Artificial Intelligence for Engineering Design, Analysis and Manufacturing
Speech and sketching: an empirical study of multimodal interaction

SBIM '07 Proceedings of the 4th Eurographics workshop on Sketch-based interfaces and modeling
Robust gesture processing for multimodal interaction

ICMI '08 Proceedings of the 10th international conference on Multimodal interfaces
Middleware mechanisms for supporting multimodal interactions in smart environments

Computer Communications
Review: Ambient intelligence: Technologies, applications, and opportunities

Pervasive and Mobile Computing
Do predictions of visual perception aid design?

ACM Transactions on Applied Perception (TAP)
CogSketch: open-domain sketch understanding for cognitive science research and for education

SBM'08 Proceedings of the Fifth Eurographics conference on Sketch-Based Interfaces and Modeling
A Dynamic Spoken Dialogue Interface for Ambient Intelligence Interaction

International Journal of Ambient Computing and Intelligence

Quantified Score

Hi-index	0.00

Visualization

Abstract

While sketches are commonly and effectively used in the early stages of design, some information is far more easily conveyed verbally than by sketching. In response, we have combined sketching with speech, enabling a more natural form of communication. We studied the behavior of people sketching and speaking, and from this derived a set of rules for segmenting and aligning the signals from both modalities. Once the inputs are aligned, we use both modalities in interpretation. The result is a more natural interface to our system.