Exploiting sensor fusion architectures and stimuli complementarity in AV speech recognition

Robert-Ribes, Jordi; Piquemal, Michel; Schwartz, Jean‐Luc; Escudier, Pierre

doi:10.1007/978-3-662-13015-5_14

Cited by 18 publications

(16 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multimodal recognition can improve performance compared with unimodal recognition by utilizing complementary sources of information [9,15,42]. Multimodal integration is commonly achieved by two different approaches.…”

Section: Audio-visual Integration Mechanismsmentioning

confidence: 99%

Audio-visual speech recognition using deep learning

et al. 2014

View full text Add to dashboard Cite

Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for reliable speech recognition, particularly when the audio is corrupted by noise. However, cautious selection of sensory features is crucial for attaining high recognition performance. In the machine-learning community, deep learning approaches have recently attracted increasing attention because deep neural networks can effectively extract robust latent features that enable various recognition algorithms to demonstrate revolutionary generalization capabilities under diverse application conditions. This study introduces a connectionist-hidden Markov model (HMM) system for noise-robust AVSR. First, a deep denoising autoencoder is utilized for acquiring noise-robust audio features. By preparing the training data for the network with pairs of consecutive multiple steps of deteriorated audio features and the corresponding clean features, the network is trained to output denoised audio features from the corresponding features deteriorated by noise. Second, a convolutional neural network (CNN) is utilized to extract visual features from raw mouth area images. By preparing the training data for the CNN as pairs of raw images and the corresponding phoneme label outputs, the network is trained to predict phoneme labels from the corresponding mouth area input images. Finally, a multi-stream HMM (MSHMM) is applied for integrating the acquired audio and visual HMMs independently trained with the respective features. By comparing the cases when normal and denoised mel-frequency cepstral coefficients (MFCCs) are utilized as audio features to the HMM, our unimodal isolated word recognition results demonstrate that approximately 65 % word recognition rate gain is attained with denoised MFCCs under 10 dB signalto-noise-ratio (SNR) for the audio signal input. Moreover, our multimodal isolated word recognition results utilizing MSHMM with denoised MFCCs and acquired visual features demonstrate that an additional word recognition rate gain is attained for the SNR conditions below 10 dB.

show abstract

Section: Audio-visual Integration Mechanismsmentioning

confidence: 99%

Audio-visual speech recognition using deep learning

et al. 2014

View full text Add to dashboard Cite

show abstract

“…Using combined audio and visual features, recognition performance was improved by a maximum of 10% at high and low SNR's over an audio-only recogniser. Future work will focus on finding more effective ways of combining the audio and visual information with the aim of ensuring that the combined performance is always at least as good as the performance using either modality [1,14,16,17] and in deriving more discriminative features from the scale histogram.…”

Section: Discussionmentioning

confidence: 99%

Audiovisual speech recognition using multiscale nonlinear image decomposition

Matthews

Bangham²,

Cox³

Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96

View full text Add to dashboard Cite

There has recently been increasing interest in the idea of enhancing speech recognition by the use of visual information derived from the face of the talker. This paper demonstrates the use of nonlinear image decomposition, in the form of a 'sieve', applied to the task of visual speech recognition. Information derived from the mouth region is used in visual and audiovisual speech recognition of a database of the letters A-Z for four talkers. A scale histogram is generated directly from the grayscale pixels of a window containing the talkers mouth on a per frame basis. Results are presented for visual-only, audio-only and in a simple audiovisual case.

show abstract

“…The previous section indicated how perceptual experiments with human observers can guide the development of computer architectures and systems design, for example, by suggesting how well visual cues are captured at different image resolutions. Cognitive studies have also suggested different architectures for the combination of the auditory and visual modalities [15].…”

Section: Audio-visual Integrationmentioning

confidence: 99%

“…Four models of audio-visual speech perception were described by Robert-Ribes et al [15], as follows:…”

Section: Models From Cognitive Psychologymentioning

confidence: 99%

Using the visual component in automatic speech recognition

Brooke

Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96

View full text Add to dashboard Cite

The movements of talkers' faces are known to convey visual cues that can improve speech intelligibility, especially where there is noise or hearing-impairment. This suggests that visible facial gestures could be exploited to enhance speech intelligibility in automatic systems. Handling the volume of data represented by images of talkers' faces implies some form of data compression. Rather than using conventional feature extraction approaches, image coding and compression can be achieved using datadriven, statistically-oriented techniques such as artificial neuralnetworks (ANNs) or principal component analysis (PCA). A major issue is the combination of the audio and visual data so that the best use can be made of the two modalities together. Perceptual experiments may offer guidance on suitable machine architectures, many of which currently use hidden Markov models (HMMs).

show abstract

Exploiting sensor fusion architectures and stimuli complementarity in AV speech recognition

Cited by 18 publications

References 0 publications

Audio-visual speech recognition using deep learning

Audio-visual speech recognition using deep learning

Audiovisual speech recognition using multiscale nonlinear image decomposition

Using the visual component in automatic speech recognition

Contact Info

Product

Resources

About