Incorporating gesture and gaze into multimodal models of human-to-human communication

Authors:
Lei Chen
Affiliations:
Purdue University, West Lafayette, IN
Venue:
NAACL-DocConsortium '06 Proceedings of the 2006 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology: companion volume: doctoral consortium
Year:
2006

Citing 8
Cited 0

Multimodal human discourse: gesture and speech

ACM Transactions on Computer-Human Interaction (TOCHI)
A Parallel Algorithm for Dynamic Gesture Tracking

RATFG-RTS '99 Proceedings of the International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems
Gesture Patterns during Speech Repairs

ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
Multimodal model integration for sentence unit detection

Proceedings of the 6th international conference on Multimodal interfaces
The virtuality continuum revisited

CHI '05 Extended Abstracts on Human Factors in Computing Systems
Structural event detection for rich transcription of speech

Structural event detection for rich transcription of speech
VACE multimodal meeting corpus

MLMI'05 Proceedings of the Second international conference on Machine Learning for Multimodal Interaction
A multimodal analysis of floor control in meetings

MLMI'06 Proceedings of the Third international conference on Machine Learning for Multimodal Interaction

Quantified Score

Hi-index	0.00

Visualization

Abstract

Structural information in language is important for obtaining a better understanding of a human communication (e.g., sentence segmentation, speaker turns, and topic segmentation). Human communication involves a variety of multimodal behaviors that signal both propositional content and structure, e.g., gesture, gaze, and body posture. These non-verbal signals have tight temporal and semantic links to spoken content. In my thesis, I am working on incorporating non-verbal cues into a multimodal model to better predict the structural events to further improve the understanding of human communication. Some research results are summarized in this document and my future research plan is described.