Intelligent multi-media interface technology
Intelligent user interfaces
Interaction techniques using hand tracking and speech recognition
Multimedia interface design
Gestures with speech for graphic manipulation
International Journal of Man-Machine Studies
Integrating simultaneous input from speech, gaze, and hand gestures
Intelligent multimedia interfaces
Research in multimedia and multimodal parsing and generation
Artificial Intelligence Review - Special issue on integration of natural language and vision processing: intelligent multimedia
A gesture processing framework for multimodal interaction in virtual reality
AFRIGRAPH '01 Proceedings of the 1st international conference on Computer graphics, virtual reality and visualisation
“Put-that-there”: Voice and gesture at the graphics interface
SIGGRAPH '80 Proceedings of the 7th annual conference on Computer graphics and interactive techniques
Avocado: A Distributed Virtual Reality Framework
VR '99 Proceedings of the IEEE Virtual Reality
Unification-based multimodal parsing
COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 1
Finite-state multimodal parsing and understanding
COLING '00 Proceedings of the 18th conference on Computational linguistics - Volume 1
ICASSP '96 Proceedings of the Acoustics, Speech, and Signal Processing, 1996. on Conference Proceedings., 1996 IEEE International Conference - Volume 06
Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality
Proceedings of the 5th international conference on Multimodal interfaces
A model-based approach for real-time embedded multimodal systems in military aircrafts
Proceedings of the 6th international conference on Multimodal interfaces
A user interface framework for multimodal VR interactions
ICMI '05 Proceedings of the 7th international conference on Multimodal interfaces
ACM Transactions on Computer-Human Interaction (TOCHI)
Fusion engines for multimodal input: a survey
Proceedings of the 2009 international conference on Multimodal interfaces
Proceedings of the 2009 international conference on Multimodal interfaces
A formal description of multimodal interaction techniques for immersive virtual reality applications
INTERACT'05 Proceedings of the 2005 IFIP TC13 international conference on Human-Computer Interaction
Knowledge in the loop: semantics representation for multimodal simulative environments
SG'05 Proceedings of the 5th international conference on Smart Graphics
Modeling ontology for multimodal interaction in ubiquitous computing systems
Proceedings of the 2012 ACM Conference on Ubiquitous Computing
Modeling multimodal integration with event logic charts
Proceedings of the 14th ACM international conference on Multimodal interaction
Formal description of multi-touch interactions
Proceedings of the 5th ACM SIGCHI symposium on Engineering interactive computing systems
| Hi-index | 0.00 |
This article presents one core component for enabling multimodal-speech and gesture-driven interaction in and for Virtual Environments. A so-called temporal Augmented Transition Network (tATN) is introduced. It allows to integrate and evaluate information from speech, gesture, and a given application context using a combined syntactic/semantic parse approach. This tATN represents the target structure for a multimodal integration markup language (MIML). MIML centers around the specification of multimodal interactions by letting an application designer declare temporal and semantic relations between given input utterance percepts and certain application states in a declarative and portable manner. A subsequent parse pass translates MIML into corresponding tATNs which are directly loaded and executed by a simulation engines scripting facility.