Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition

Sterpu, George; Saam, Christian; Harte, Naomi

doi:10.1145/3242969.3243014

Cited by 59 publications

(55 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Neural networks, such as deep neural (DNN), recurrent neural network (RNN) and long short-term memory (LSTM) have been introduced to the field of speech recognition [7][8][9] and AV-ASR [1,[10][11][12][13][14][15][16]. End-to-end neural networks are challenging the dominance of HMM as a core technology.…”

Section: Related Workmentioning

confidence: 99%

WaveNet With Cross-Attention for Audiovisual Speech Recognition

et al. 2020

View full text Add to dashboard Cite

In this paper, the WaveNet with cross-attention is proposed for AudioVisual Automatic Speech Recognition (AV-ASR) to address multimodal feature fusion and frame alignment problems between two data streams. WaveNet is usually used for speech generation and speech recognition, however, in this paper, we extent it to audiovisual speech recognition, and the cross-attention mechanism is introduced into different places of WaveNet for feature fusion. The proposed cross-attention mechanism tries to explore the correlated frames of visual feature to the acoustic feature frame. The experimental results show that the WaveNet with cross-attention can reduce the Tibetan single syllable error about 4.5% and English word error about 39.8% relative to the audio-only speech recognition, and reduce Tibetan single syllable error about 35.1% and English word error about 21.6% relative to the conventional feature concatenation method for AV-ASR. INDEX TERMS Cross-attention mechanism, multimodal speech recognition, WaveNet model, end-to-endmodel.

show abstract

Section: Related Workmentioning

confidence: 99%

WaveNet With Cross-Attention for Audiovisual Speech Recognition

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Experiments are reported on the TCD-TIMIT corpus [12], a very popular dataset in the field [18,[24][25][26][27][28][29][30][31]. The database contains audio-visual recordings of continuous speech by 62 speakers uttering 6913 phonetically-rich TIMIT sentences (6k word vocabulary) in studio-like conditions, concurrently recorded by two cameras providing frontal (0 o ) and near-frontal (30 o ) data at a 1920 × 1080-pixel resolution and 30 Hz frame-rate.…”

Section: Datasetmentioning

confidence: 99%

“…We provide additional system implementation details in Section 3, and we evaluate our developed networks in Section 4. Specifically, we study computational efficiency and VSR accuracy on the publicly available TCD-TIMIT corpus [12], a popular database for lipreading [18,[24][25][26][27][28] and other audio-visual speech processing tasks [29][30][31]. Our experiments show that our best model, a "MobiLipNetV2" with 3D pointwise convolutions, exhibits dramatically improved computational efficiency compared to both a baseline 3D-CNN and a state-of-theart ResNet, with no or minimal accuracy degradation.…”

Section: Introductionmentioning

confidence: 99%

MobiLipNet: Resource-Efficient Deep Learning Based Lipreading

Koumparoulis

Potamianos

2019

Interspeech 2019

View full text Add to dashboard Cite

Recent works in visual speech recognition utilize deep learning advances to improve accuracy. Focus however has been primarily on recognition performance, while ignoring the computational burden of deep architectures. In this paper we address these issues concurrently, aiming at both high computational efficiency and recognition accuracy in lipreading. For this purpose, we investigate the MobileNet convolutional neural network architectures, recently proposed for image classification. In addition, we extend the 2D convolutions of MobileNets to 3D ones, in order to better model the spatio-temporal nature of the lipreading problem. We investigate two architectures in this extension, introducing the temporal dimension as part of either the depthwise or the pointwise MobileNet convolutions. To further boost computational efficiency, we also consider using pointwise convolutions alone, as well as networks operating on half the mouth region. We evaluate the proposed architectures on speaker-independent visual-only continuous speech recognition on the popular TCD-TIMIT corpus. Our best system outperforms a baseline CNN by 4.27% absolute in word error rate and over 12 times in computational efficiency, whereas, compared to a state-of-the-art ResNet, it is 37 times more efficient at a minor 0.07% absolute error rate degradation.

show abstract

“…Noda [24] makes use of CNN to extract visual feature and combines it with audio feature by a multi-stream HMM. Feature fusion and then using a one stream model is the dominant approach [25,20,26] in AVSR. [26] correlates every frame of acoustic feature with visual context feature acquired by cross modality attention.…”

Section: Introductionmentioning

confidence: 99%

“…Feature fusion and then using a one stream model is the dominant approach [25,20,26] in AVSR. [26] correlates every frame of acoustic feature with visual context feature acquired by cross modality attention. The correlated feature is then decoded by an attention based decoder.…”

Section: Introductionmentioning

confidence: 99%

Modality Attention for End-to-end Audio-visual Speech Recognition

Zhou

Yang²,

Chen³

et al. 2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for robust speech recognition, especially in noisy environment. In this paper, we propose a novel multimodal attention based method for audio-visual speech recognition which could automatically learn the fused representation from both modalities based on their importance. Our method is realized using state-of-the-art sequence-to-sequence (Seq2seq) architectures. Experimental results show that relative improvements from 2% up to 36% over the auditory modality alone are obtained depending on the different signal-to-noise-ratio (SNR). Compared to the traditional feature concatenation methods, our proposed approach can achieve better recognition performance under both clean and noisy conditions. We believe modality attention based end-to-end method can be easily generalized to other multimodal tasks with correlated information.

show abstract

Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition

Cited by 59 publications

References 18 publications

WaveNet With Cross-Attention for Audiovisual Speech Recognition

WaveNet With Cross-Attention for Audiovisual Speech Recognition

MobiLipNet: Resource-Efficient Deep Learning Based Lipreading

Modality Attention for End-to-end Audio-visual Speech Recognition

Contact Info

Product

Resources

About