Algorithm schemata and data structures in syntactic processing
Readings in natural language processing
Information-based syntax and semantics: Vol. 1: fundamentals
Information-based syntax and semantics: Vol. 1: fundamentals
Building visual language parsers
CHI '91 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
CHI '91 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Intelligent multi-media interface technology
Intelligent user interfaces
The logic of typed feature structures
The logic of typed feature structures
Integrating simultaneous input from speech, gaze, and hand gestures
Intelligent multimedia interfaces
Multimodal interfaces for dynamic interactive maps
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Integration and synchronization of input modes during multimodal human-computer interaction
Proceedings of the ACM SIGCHI Conference on Human factors in computing systems
QuickSet: multimodal interaction for distributed applications
MULTIMEDIA '97 Proceedings of the fifth ACM international conference on Multimedia
An efficient context-free parsing algorithm
Communications of the ACM
Finite-State Language Processing
Finite-State Language Processing
“Put-that-there”: Voice and gesture at the graphics interface
SIGGRAPH '80 Proceedings of the 7th annual conference on Computer graphics and interactive techniques
Unification-based multimodal integration
ACL '98 Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics
Confirmation in multimodal systems
COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 2
F-PATR: functional constraints for unification-based grammars
ACL '93 Proceedings of the 31st annual meeting on Association for Computational Linguistics
ICASSP '96 Proceedings of the Acoustics, Speech, and Signal Processing, 1996. on Conference Proceedings., 1996 IEEE International Conference - Volume 06
Something from nothing: augmenting a paper-based work practice via multimodal interaction
DARE '00 Proceedings of DARE 2000 on Designing augmented reality environments
Creating tangible interfaces by augmenting physical objects with multimodal language
Proceedings of the 6th international conference on Intelligent user interfaces
Comparing paper and tangible, multimodal tools
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Multimodal event parsing for intelligent user interfaces
Proceedings of the 8th international conference on Intelligent user interfaces
Smart Classroom - An Intelligent Environment for Tele-education
PCM '01 Proceedings of the Second IEEE Pacific Rim Conference on Multimedia: Advances in Multimedia Information Processing
Advances in the robust processing of multimodal speech and pen systems
Multimodal interface for human-machine communication
Designing Transition Networks for Multimodal VR-Interactions Using a Markup Language
ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
A Map-Based System Using Speech and 3D Gestures for Pervasive Computing
ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
Referring to Objects with Spoken and Haptic Modalities
ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
Context-Based Multimodal Input Understanding in Conversational Systems
ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality
Proceedings of the 5th international conference on Multimodal interfaces
Architecture and implementation of multimodal plug and play
Proceedings of the 5th international conference on Multimodal interfaces
A probabilistic approach to reference resolution in multimodal user interfaces
Proceedings of the 9th international conference on Intelligent user interfaces
A visual modality for the augmentation of paper
Proceedings of the 2001 workshop on Perceptive user interfaces
Deixis and conjunction in multimodal systems
COLING '00 Proceedings of the 18th conference on Computational linguistics - Volume 1
Finite-state multimodal parsing and understanding
COLING '00 Proceedings of the 18th conference on Computational linguistics - Volume 1
Proceedings of the 6th international conference on Multimodal interfaces
Linguistic theories in efficient multimodal reference resolution: an empirical investigation
Proceedings of the 10th international conference on Intelligent user interfaces
Finite-state multimodal integration and understanding
Natural Language Engineering
Semantics-based representation for multimodal interpretation in conversational systems
COLING '02 Proceedings of the 19th international conference on Computational linguistics - Volume 1
Integrating sketch and speech inputs using spatial information
ICMI '05 Proceedings of the 7th international conference on Multimodal interfaces
A user interface framework for multimodal VR interactions
ICMI '05 Proceedings of the 7th international conference on Multimodal interfaces
Coordination and context-dependence in the generation of embodied conversation
INLG '00 Proceedings of the first international conference on Natural language generation - Volume 14
Salience modeling based on non-verbal modalities for spoken language understanding
Proceedings of the 8th international conference on Multimodal interfaces
Multimodal fusion: a new hybrid strategy for dialogue systems
Proceedings of the 8th international conference on Multimodal interfaces
Optimization in multimodal interpretation
ACL '04 Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics
A salience driven approach to robust input interpretation in multimodal conversational systems
HLT '05 Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing
A novel method for multi-sensory data fusion in multimodal human computer interaction
OZCHI '06 Proceedings of the 18th Australia conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments
Fusion of children's speech and 2D gestures when conversing with 3D characters
Signal Processing - Special section: Multimodal human-computer interfaces
Signal Processing - Special section: Multimodal human-computer interfaces
An efficient unification-based multimodal language processor in multimodal input fusion
OZCHI '07 Proceedings of the 19th Australasian conference on Computer-Human Interaction: Entertaining User Interfaces
Proceedings of the 13th international conference on Intelligent user interfaces
References to graphical objects in interactive multimodal queries
Knowledge-Based Systems
An integrative recognition method for speech and gestures
ICMI '08 Proceedings of the 10th international conference on Multimodal interfaces
Context shifts: extending the meanings of physical objects with language
Human-Computer Interaction
Clavius: bi-directional parsing for generic multimodal interaction
COLING ACL '06 Proceedings of the 21st International Conference on computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop
HLT-NAACL-Short '04 Proceedings of HLT-NAACL 2004: Short Papers
UI on the Fly: generating a multimodal user interface
HLT-NAACL-Short '04 Proceedings of HLT-NAACL 2004: Short Papers
Robust understanding in multimodal interfaces
Computational Linguistics
Cognitive principles in robust multimodal interpretation
Journal of Artificial Intelligence Research
Multimodal interaction: a new focal area for AI
IJCAI'01 Proceedings of the 17th international joint conference on Artificial intelligence - Volume 2
IEEE Transactions on Audio, Speech, and Language Processing - Special issue on multimodal processing in speech-based interactions
Usage patterns and latent semantic analyses for task goal inference of multimodal user interactions
Proceedings of the 15th international conference on Intelligent user interfaces
An input-parsing algorithm supporting integration of deictic gesture in natural language interface
HCI'07 Proceedings of the 12th international conference on Human-computer interaction: intelligent multimodal interaction environments
A general framework for incremental processing of multimodal inputs
ICMI '11 Proceedings of the 13th international conference on multimodal interfaces
Review Article: Multimodal interaction: A review
Pattern Recognition Letters
Latent Semantic Analysis for Multimodal User Input With Speech and Gestures
IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP)
| Hi-index | 0.00 |
In order to realize their full potential, multimodal systems need to support not just input from multiple modes, but also synchronized integration of modes. Johnston et al (1997) model this integration using a unification operation over typed feature structures. This is an effective solution for a broad class of systems, but limits multimodal utterances to combinations of a single spoken phrase with a single gesture. We show how the unification-based approach can be scaled up to provide a full multimodal grammar formalism. In conjunction with a multidimensional chart parser, this approach supports integration of multiple elements distributed across the spatial, temporal, and acoustic dimensions of multimodal interaction. Integration strategies are stated in a high level unification-based rule formalism supporting rapid prototyping and iterative development of multimodal systems.