Speech recognition by integrating audio, visual and contextual features based on neural networks

Authors:
Myung Won Kim;Joung Woo Ryu;Eun Ju Kim
Affiliations:
School of Computing, Soongsil University, Seoul, Korea;School of Computing, Soongsil University, Seoul, Korea;School of Computing, Soongsil University, Seoul, Korea
Venue:
ICNC'05 Proceedings of the First international conference on Advances in Natural Computation - Volume Part II
Year:
2005

Citing 2
Cited 0

Multi-Source Neural Networks for Speech Recognition: A Review of Recent Results

IJCNN '00 Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks (IJCNN'00)-Volume 5 - Volume 5
A review of speech-based bimodal recognition

IEEE Transactions on Multimedia

Quantified Score

Hi-index	0.00

Visualization

Abstract

Recent researches have been focusing on fusion of audio and visual features for reliable speech recognition in noisy environments. In this paper, we propose a neural network based model of robust speech recognition by integrating audio, visual, and contextual information. Bimodal Neural Network (BMNN) is a multi-layer perceptron of 4 layers, which combines audio and visual features of speech to compensate loss of audio information caused by noise. In order to improve the accuracy of speech recognition in noisy environments, we also propose a post-processing based on contextual information which are sequential patterns of words spoken by a user. Our experimental results show that our model outperforms any single mode models. Particularly, when we use the contextual information, we can obtain over 90% recognition accuracy even in noisy environments, which is a significant improvement compared with the state of art in speech recognition.