Intelligent multi-media interface technology
Intelligent user interfaces
An object-oriented 3D graphics toolkit
SIGGRAPH '92 Proceedings of the 19th annual conference on Computer graphics and interactive techniques
Gestures with speech for graphic manipulation
International Journal of Man-Machine Studies
Integrating simultaneous input from speech, gaze, and hand gestures
Intelligent multimedia interfaces
Research in multimedia and multimodal parsing and generation
Artificial Intelligence Review - Special issue on integration of natural language and vision processing: intelligent multimedia
The virtual human as a multimodal interface
AVI '00 Proceedings of the working conference on Advanced visual interfaces
EGVE '02 Proceedings of the workshop on Virtual environments 2002
A gesture processing framework for multimodal interaction in virtual reality
AFRIGRAPH '01 Proceedings of the 1st international conference on Computer graphics, virtual reality and visualisation
Multimodal Interaction for 2D and 3D Environments
IEEE Computer Graphics and Applications
The human-computer interaction handbook
“Put-that-there”: Voice and gesture at the graphics interface
SIGGRAPH '80 Proceedings of the 7th annual conference on Computer graphics and interactive techniques
Avocado: A Distributed Virtual Reality Framework
VR '99 Proceedings of the IEEE Virtual Reality
Designing Transition Networks for Multimodal VR-Interactions Using a Markup Language
ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
Referring to Objects with Spoken and Haptic Modalities
ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
A Multi-Modal Interface for an Interactive Simulated Vascular Reconstruction System
ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
Modular Approach of Multimodal Integration in a Virtual Environment
ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality
Proceedings of the 5th international conference on Multimodal interfaces
Using multimodal interaction to navigate in arbitrary virtual VRML worlds
Proceedings of the 2001 workshop on Perceptive user interfaces
Unification-based multimodal integration
ACL '98 Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
Unification-based multimodal parsing
COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 1
Resolving Object References in Multimodal Dialogues for Immersive Virtual Environments
VR '04 Proceedings of the IEEE Virtual Reality 2004
Automatic Data Exchange and Synchronization for Knowledge-Based Intelligent Virtual Environments
VR '05 Proceedings of the 2005 IEEE Conference 2005 on Virtual Reality
ICASSP '96 Proceedings of the Acoustics, Speech, and Signal Processing, 1996. on Conference Proceedings., 1996 IEEE International Conference - Volume 06
Towards the integration of shape-related information in 3-D gestures and speech
Proceedings of the 8th international conference on Multimodal interfaces
A novel method for multi-sensory data fusion in multimodal human computer interaction
OZCHI '06 Proceedings of the 18th Australia conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments
AFRIGRAPH '07 Proceedings of the 5th international conference on Computer graphics, virtual reality, visualisation and interaction in Africa
An efficient unification-based multimodal language processor in multimodal input fusion
OZCHI '07 Proceedings of the 19th Australasian conference on Computer-Human Interaction: Entertaining User Interfaces
An input-parsing algorithm supporting integration of deictic gesture in natural language interface
HCI'07 Proceedings of the 12th international conference on Human-computer interaction: intelligent multimodal interaction environments
Insights on the design of intml
Presence: Teleoperators and Virtual Environments
A multimodal reference resolution approach in virtual environment
VSMM'06 Proceedings of the 12th international conference on Interactive Technologies and Sociotechnical Systems
Making the mainstream accessible: what's in a game?
ICCHP'06 Proceedings of the 10th international conference on Computers Helping People with Special Needs
Location-Based interaction with children for edutainment
PIT'06 Proceedings of the 2006 international tutorial and research conference on Perception and Interactive Technologies
3D Virtual worlds and the metaverse: Current status and future possibilities
ACM Computing Surveys (CSUR)
| Hi-index | 0.00 |
This article presents a User Interface (UI) framework for multimodal interactions targeted at immersive virtual environments. Its configurable input and gesture processing components provide an advanced behavior graph capable of routing continuous data streams asynchronously. The framework introduces a Knowledge Representation Layer which augments objects of the simulated environment with Semantic Entities as a central object model that bridges and interfaces Virtual Reality (VR) and Artificial Intelligence (AI) representations. Specialized node types use these facilities to implement required processing tasks like gesture detection, preprocessing of the visual scene for multimodal integration, or translation of movements into multimodally initialized gestural interactions. A modified Augmented Transition Nettwork (ATN) approach accesses the knowledge layer as well as the preprocessing components to integrate linguistic, gestural, and context information in parallel. The overall framework emphasizes extensibility, adaptivity and reusability, e.g., by utilizing persistent and interchangeable XML-based formats to describe its processing stages.