A Modality Replacement Framework for the Communication between Blind and Hearing Impaired People

Authors:
Konstantinos Moustakas;Dimitrios Tzovaras;Laila Dybkjær;Niels Ole Bernsen
Affiliations:
Informatics and Telematics Institute, Centre for Research and Technology Hellas, Thermi-Thessaloniki, Greece 60361;Informatics and Telematics Institute, Centre for Research and Technology Hellas, Thermi-Thessaloniki, Greece 60361;NISLab, Copenhagen, Denmark;NISLab, Copenhagen, Denmark
Venue:
UAHCI '09 Proceedings of the 5th International Conference on Universal Access in Human-Computer Interaction. Part III: Applications and Services
Year:
2009

Citing 5
Cited 0

A paradigm shift: alternative interaction techniques for use with mobile & wearable devices

CASCON '03 Proceedings of the 2003 conference of the Centre for Advanced Studies on Collaborative research
SQ-Map: Efficient Layered Collision Detection and Haptic Rendering

IEEE Transactions on Visualization and Computer Graphics
Haptic Rendering of Visual Data for the Visually Impaired

IEEE MultiMedia
Transforming 3D coloured pixels into musical instrument notes for vision substitution applications

Journal on Image and Video Processing
Recognizing two handed gestures with generative, discriminative and ensemble methods via fisher kernels

MRCS'06 Proceedings of the 2006 international conference on Multimedia Content Representation, Classification and Security

Quantified Score

Hi-index	0.00

Visualization

Abstract

This paper presents a multimodal framework for the communication between blind and hearing impaired people. The algorithms that are developed are based on the concept of modality replacement, which is the use of information originating from various modalities to compensate for the missing input modality of the system or the users. Spatial information is conveyd in the blind user's terminal through the haptic modality utilizing an efficient haptic rendering scheme, while verbal information is presented in the hearing impaired user's terminal through sign language synthesis. All technologies are integrated in a virtual treasure hunting game, where a blind and a hearing impaired user have to collaborate so as to navigate in the virtual environment and solve the riddle of the game. Usability evaluation of this framework has shown significant impact of the proposed system for the disabled users.