A Multimedia System for Temporally Situated Perceptual Psycholinguistic Analysis

Authors:
Francis Quek;Robert Bryll;Cemil Kirbas;Hasan Arslan;David McNeill
Affiliations:
Vision Interfaces and Systems Laboratory, Department of Computer Science and Engineering, Wright State University, Dayton, OH, USA. francis.quek@wright.edu;Vision Interfaces and Systems Laboratory, Department of Computer Science and Engineering, Wright State University, Dayton, OH, USA. bryll.2@wright.edu;Vision Interfaces and Systems Laboratory, Department of Computer Science and Engineering, Wright State University, Dayton, OH, USA. kirbas.2@wright.edu;Vision Interfaces and Systems Laboratory, Department of Computer Science and Engineering, Wright State University, Dayton, OH, USA;Departments of Psychology and Linguistics, University of Chicago, Chicago, IL, USA.
Venue:
Multimedia Tools and Applications
Year:
2002

Citing 3
Cited 3

Principles and guidelines in software user interface design

Principles and guidelines in software user interface design
Retrieving and visualizing video

Communications of the ACM
Gesture Cues for Conversational Interaction in Monocular Video

RATFG-RTS '99 Proceedings of the International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems

Multimodal human discourse: gesture and speech

ACM Transactions on Computer-Human Interaction (TOCHI)
MacVisSTA: a system for multimodal analysis

Proceedings of the 6th international conference on Multimodal interfaces
The catchment feature model: a device for multimodal fusion and a bridge between signal and sense

EURASIP Journal on Applied Signal Processing

Quantified Score

Hi-index	0.00

Visualization

Abstract

Perceptual analysis of video (analysis by unaided ear and eye) plays an important role in such disciplines as psychology, psycholinguistics, linguistics, anthropology, and neurology. In the specific domain of psycholinguistic analysis of gesture and speech, researchers micro-analyze videos of subjects using a high quality video cassette recorder that has a digital freeze capability down to the specific frame. Such analyses are very labor intensive and slow. We present a multimedia system for perceptual analysis of video data using a multiple, dynamically linked representation model. The system components are linked through a time portal with a current time focus. The system provides mechanisms to analyze overlapping hierarchical interpretations of the discourse, and integrates visual gesture analysis, speech analysis, video gaze analysis, and text transcription into a coordinated whole. The various interaction components facilitate accurate multi-point access to the data. While this system is currently used to analyze gesture, speech and gaze in human discourse, the system described may be applied to any other field where careful analysis of temporal synchronies in video is important.