Speech-Enabled Augmented Reality Supporting Mobile Industrial Maintenance
IEEE Pervasive Computing
Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality
Proceedings of the 5th international conference on Multimodal interfaces
"Move the couch where?": developing an augmented reality multimodal interface
ISMAR '06 Proceedings of the 5th IEEE and ACM International Symposium on Mixed and Augmented Reality
Experiences with Handheld Augmented Reality
ISMAR '07 Proceedings of the 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality
ARSC: Augmented reality student card
Computers & Education
Just say it: an evaluation of speech interfaces for augmented reality design applications
AICS'09 Proceedings of the 20th Irish conference on Artificial intelligence and cognitive science
SYNASC '10 Proceedings of the 2010 12th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing
Audio-visual speech modeling for continuous speech recognition
IEEE Transactions on Multimedia
Learning OpenCV: Computer Vision in C++ with the OpenCV Library
Learning OpenCV: Computer Vision in C++ with the OpenCV Library
Hi-index | 0.00 |
Many recent studies show that Augmented Reality (AR) and Automatic Speech Recognition (ASR) technologies can be used to help people with disabilities. Many of these studies have been performed only in their specialized field. Audio-Visual Speech Recognition (AVSR) is one of the advances in ASR technology that combines audio, video, and facial expressions to capture a narrator's voice. In this paper, we combine AR and AVSR technologies to make a new system to help deaf and hard-of-hearing people. Our proposed system can take a narrator's speech instantly and convert it into a readable text and show the text directly on an AR display. Therefore, in this system, deaf people can read the narrator's speech easily. In addition, people do not need to learn sign-language to communicate with deaf people. The evaluation results show that this system has lower word error rate compared to ASR and VSR in different noisy conditions. Furthermore, the results of using AVSR techniques show that the recognition accuracy of the system has been improved in noisy places. Also, the results of a survey that was conducted with 100 deaf people show that more than 80 % of deaf people are very interested in using our system as an assistant in portable devices to communicate with people.