The syntactic process
Maintaining knowledge about temporal intervals
Communications of the ACM
Finite-state multimodal parsing and understanding
COLING '00 Proceedings of the 18th conference on Computational linguistics - Volume 1
Multi-Modal Definite Clause Grammar
COLING '94 Proceedings of the 15th conference on Computational linguistics - Volume 2
Coupling CCG and hybrid logic dependency semantics
ACL '02 Proceedings of the 40th Annual Meeting on Association for Computational Linguistics
Recognition of Composite Human Activities through Context-Free Grammar Based Representation
CVPR '06 Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2
Human-Robot dialogue for joint construction tasks
Proceedings of the 8th international conference on Multimodal interfaces
MultiML: a general purpose representation language for multimodal human utterances
ICMI '08 Proceedings of the 10th international conference on Multimodal interfaces
Attention driven visual processing for an interactive dialog robot
Proceedings of the 2009 ACM symposium on Applied Computing
| Hi-index | 0.00 |
Multimodal systems must process several input streams efficiently and represent the input in a way that allows the establishment of connections between modalities. This paper describes a multimodal system that uses Combinatory Categorial Grammars to parse several input streams and translate them into logical formulas. These logical formulas are expressed in Hybrid Logic, which is very suitable for multimodal integration because it can represent temporal relationships between modes in an abstract way. This level of abstraction makes it possible to define rules for multimodal processing in a straightforward way.