Synergistic use of direct manipulation and natural language
CHI '89 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
A synthetic visual environment with hand gesturing and voice input
CHI '89 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Intelligent multi-media interface technology
Intelligent user interfaces
Designing the user interface (2nd ed.): strategies for effective human-computer interaction
Designing the user interface (2nd ed.): strategies for effective human-computer interaction
The role of natural language in a multimodal interface
UIST '92 Proceedings of the 5th annual ACM symposium on User interface software and technology
Architectural qualities and principles for multimodal and multimedia interfaces
Multimedia interface design
Knowledge-based augmented reality
Communications of the ACM - Special issue on computer augmented environments: back to the real world
Gestures with speech for graphic manipulation
International Journal of Man-Machine Studies
Integrating simultaneous input from speech, gaze, and hand gestures
Intelligent multimedia interfaces
Semantic Structures: Advances in Natural Language Processing
Semantic Structures: Advances in Natural Language Processing
In-Depth Understanding: A Computer Model of Integrated Processing for Narrative Comprehension
In-Depth Understanding: A Computer Model of Integrated Processing for Narrative Comprehension
Conceptual Information Processing
Conceptual Information Processing
“Put-that-there”: Voice and gesture at the graphics interface
SIGGRAPH '80 Proceedings of the 7th annual conference on Computer graphics and interactive techniques
An application of shared virtual reality to situational training
VRAIS '95 Proceedings of the Virtual Reality Annual International Symposium (VRAIS'95)
Constrained 3D navigation with 2D controllers
VIS '97 Proceedings of the 8th conference on Visualization '97
Integrating pedagogical capabilities in a virtual environment agent
AGENTS '97 Proceedings of the first international conference on Autonomous agents
A framework for assisted exploration with collaboration
VIS '99 Proceedings of the conference on Visualization '99: celebrating ten years
Education and Information Technologies
Constrained Navigation Environments
Dagstuhl '97, Scientific Visualization
Extending virtual humans to support team training in virtual reality
Exploring artificial intelligence in the new millennium
Integrating Pedagogical Agents into Virtual Environments
Presence: Teleoperators and Virtual Environments
Presence: Teleoperators and Virtual Environments
| Hi-index | 0.00 |
Virtual reality has made computer interfaces more intuitive but not more intelligent. This paper shows how a rule-based expert system can be combined with multimodal input and natural language techniques to add intelligence to the interface. In this way, virtual worlds can be built which recognize and respond intelligently to user actions. Coupling voice, gesture, body position and context together with an expert system creates an interface more powerful than any of these elements alone; one which establishes a true two-way dialog between human and machine. To illustrate these techniques, we present two prototype systems that allow intuitive multimodal communication within intelligent virtual environments.