Enabling multimodal communications for enhancing the ability of learning for the visually impaired
Proceedings of the 8th international conference on Multimodal interfaces
Fun to develop embodied skill: how games help the blind to understand pointing
Proceedings of the 3rd International Conference on PErvasive Technologies Related to Assistive Environments
Enabling multimodal discourse for the blind
International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction
Enabling the blind to see gestures
ACM Transactions on Computer-Human Interaction (TOCHI) - Special issue on the theory and practice of embodied interaction in HCI and interaction design
Hi-index | 0.00 |
Mathematics instruction and discourse typically involve two modes of communication: speech and graphical presentation. For the communication to remain situated, dynamic synchrony must be maintained between the speech and dynamic focus in the graphics. In normals, vision is used for two purposes: access to graphical material and awareness of embodied behavior. This embodiment awareness keeps communication situated with visual material and speech. Our goal is to assist blind students in the access to such instruction/communication. We employ the typical approach of sensory replacement for the missing visual sense. Haptic fingertip reading can replace visual material. For the embodied portion of the communication, we want to make the blind student aware of the deictic gestures performed by the teacher over the graphic in conjunction with speech. We propose the use of haptic gloves paired with computer vision based tracking to help blind students maintain reading focus on a raised line representation of a graphical presentation to which the instructor points while speaking. In this initial phase of our research, we conducted three experiments that show that: 1) The gloves convey sense of direction; 2) The gloves do not interfere in fingertip reading; 3) A person can navigate with the help of this system while listening to a story; 4) It is possible to fuse the information received from both modes. We discuss these findings in this paper.