An efficient unification-based multimodal language processor in multimodal input fusion

Authors:
Yong Sun;Yu Shi;Fang Chen;Vera Chung
Affiliations:
National ICT Australia, Eveleigh NSW, Australia and The University of Sydney, Redfern NSW, Australia;National ICT Australia, Eveleigh NSW, Australia;National ICT Australia, Eveleigh NSW, Australia and The University of Sydney, Redfern NSW, Australia;The University of Sydney, Redfern NSW, Australia
Venue:
OZCHI '07 Proceedings of the 19th Australasian conference on Computer-Human Interaction: Entertaining User Interfaces
Year:
2007

Citing 10
Cited 3

Mutual disambiguation of recognition errors in a multimodel architecture

Proceedings of the SIGCHI conference on Human Factors in Computing Systems
The syntactic process

The syntactic process
An efficient context-free parsing algorithm

Communications of the ACM
Unification-based multimodal parsing

COLING '98 Proceedings of the 17th international conference on Computational linguistics - Volume 1
Polynomial time parsing of Combinatory Categorial Grammars

ACL '90 Proceedings of the 28th annual meeting on Association for Computational Linguistics
A multimodal learning interface for sketch, speak and point creation of a schedule chart

Proceedings of the 6th international conference on Multimodal interfaces
A user interface framework for multimodal VR interactions

ICMI '05 Proceedings of the 7th international conference on Multimodal interfaces
Multimodal fusion: a new hybrid strategy for dialogue systems

Proceedings of the 8th international conference on Multimodal interfaces
An Efficient Multimodal Language Processor for Parallel Input Strings in Multimodal Input Fusion

ICSC '07 Proceedings of the International Conference on Semantic Computing
Clavius: bi-directional parsing for generic multimodal interaction

COLING ACL '06 Proceedings of the 21st International Conference on computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop

The hinge between input and output: understanding the multimodal input fusion results in an agent-based multimodal presentation system

CHI '08 Extended Abstracts on Human Factors in Computing Systems
Skipping spare information in multimodal inputs during multimodal input fusion

Proceedings of the 14th international conference on Intelligent user interfaces
Building a Practical Multimodal System with a Multimodal Fusion Module

Proceedings of the 13th International Conference on Human-Computer Interaction. Part II: Novel Interaction Methods and Techniques

Quantified Score

Hi-index	0.00

Visualization

Abstract

A Multimodal User Interface (MMUI) allows a user to interact with a computer in a way similar to human-to-human communication, for example, through speech and gesture. Being an essential component in MMUIs, Multimodal Input Fusion should be able to find the semantic interpretation of a user's intention from recognized multimodal symbols which are semantically complementary. We enhanced our efficient unification-based multimodal parsing processor, which has the potential to achieve low polynomial computational complexity while parsing versatile multimodal inputs within a speech and gesture based MMUI, to handle multimodal inputs from more than two modalities. Its ability to disambiguate speech recognition results with gesture recognition results was verified with an experiment. The analysis of experiment results demonstrates that the improvement is significant after applying this technique.