A framework for continuous multimodal sign language recognition

Authors:
Daniel Kelly;Jane Reilly Delannoy;John Mc Donald;Charles Markham
Affiliations:
N.U.I. Maynooth, Maynooth, Co. Kildare, Ireland;N.U.I. Maynooth, Maynooth, Co. Kildare, Ireland;N.U.I. Maynooth, Maynooth, Co. Kildare, Ireland;N.U.I. Maynooth, Maynooth, Co. Kildare, Ireland
Venue:
Proceedings of the 2009 international conference on Multimodal interfaces
Year:
2009

Citing 23
Cited 2

Recognition of sign language motion images

Pattern Recognition
Sign language recognition using model-based tracking and a 3D Hopfield neural network

Machine Vision and Applications
Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video

IEEE Transactions on Pattern Analysis and Machine Intelligence
An HMM-Based Threshold Model Approach for Gesture Recognition

IEEE Transactions on Pattern Analysis and Machine Intelligence
Appearance-based hand sign recognition from intensity image sequences

Computer Vision and Image Understanding
A framework for recognizing the simultaneous aspects of American sign language

Computer Vision and Image Understanding - Modeling people toward vision-based underatanding of a person's shape, appearance, and movement
Extraction of 2D Motion Trajectories and Its Application to Hand Gesture Recognition

IEEE Transactions on Pattern Analysis and Machine Intelligence
Video-Based Sign Language Recognition Using Hidden Markov Models

Proceedings of the International Gesture Workshop on Gesture and Sign Language in Human-Computer Interaction
The Recognition Algorithm with Non-contact for Japanese Sign Language Using Morphological Analysis

Proceedings of the International Gesture Workshop on Gesture and Sign Language in Human-Computer Interaction
Vision-Based Gesture Recognition: A Review

GW '99 Proceedings of the International Gesture Workshop on Gesture-Based Communication in Human-Computer Interaction
Towards an Automatic Sign Language Recognition System Using Subunits

GW '01 Revised Papers from the International Gesture Workshop on Gesture and Sign Languages in Human-Computer Interaction
A Real-Time Continuous Gesture Recognition System for Sign Language

FG '98 Proceedings of the 3rd. International Conference on Face & Gesture Recognition
A Method for Recognizing a Sequence of Sign Language Words Represented in a Japanese Sign Language Sentence

FG '00 Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000
Partly-Hidden Markov Model and its Application to Gesture Recognition

ICASSP '97 Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '97) -Volume 4 - Volume 4
An Approach Based on Phonemes to Large Vocabulary Chinese Sign Language Recognition

FGR '02 Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition
Recognition of Local Features for Camera-Based Sign Language Recognition System

ICPR '00 Proceedings of the International Conference on Pattern Recognition - Volume 4
Automatic Sign Language Analysis: A Survey and the Future beyond Lexical Meaning

IEEE Transactions on Pattern Analysis and Machine Intelligence
Rapid Signer Adaptation for Isolated Sign Language Recognition

CVPRW '06 Proceedings of the 2006 Conference on Computer Vision and Pattern Recognition Workshop
Recent developments in visual sign language recognition

Universal Access in the Information Society
Facial movement analysis in ASL

Universal Access in the Information Society
Analysis of Sign Language Gestures Using Size Functions and Principal Component Analysis

IMVIP '08 Proceedings of the 2008 International Machine Vision and Image Processing Conference
Modelling and recognition of the linguistic components in American Sign Language

Image and Vision Computing
Recognizing Spatiotemporal Gestures and Movement Epenthesis in Sign Language

IMVIP '09 Proceedings of the 2009 13th International Machine Vision and Image Processing Conference

Automatic recognition of sign language subwords based on portable accelerometer and EMG sensors

International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction
Gesture vs. gesticulation: a test protocol

HCI'13 Proceedings of the 15th international conference on Human-Computer Interaction: interaction modalities and techniques - Volume Part IV

Quantified Score

Hi-index	0.00

Visualization

Abstract

We present a multimodal system for the recognition of manual signs and non-manual signals within continuous sign language sentences. In sign language, information is mainly conveyed through hand gestures (Manual Signs). Non-manual signals, such as facial expressions, head movements, body postures and torso movements, are used to express a large part of the grammar and some aspects of the syntax of sign language. In this paper we propose a multichannel HMM based system to recognize manual signs and non-manual signals. We choose a single non-manual signal, head movement, to evaluate our framework when recognizing non-manual signals. Manual signs and non-manual signals are processed independently using continuous multidimensional HMMs and a HMM threshold model. Experiments conducted demonstrate that our system achieved a detection ratio of 0.95 and a reliability measure of 0.93.