Audio-Visual Multi-Channel Speech Separation, Dereverberation and Recognition

Abstract

Accurate recognition of cocktail party speech containing overlapping speakers, noise and reverberation remains a highly challenging task to date. Motivated by the invariance of visual modality to acoustic signal corruption, an audio-visual multi-channel speech separation, dereverberation and recognition approach featuring a full incorporation of visual information into all system components is proposed in this paper. The efficacy of the video input is consistently demonstrated in maskbased MVDR speech separation, DNN-WPE or spectral mapping (SpecM) based speech dereverberation front-end and Conformer ASR back-end. Audio-visual integrated front-end architectures performing speech separation and dereverberation in a pipelined or joint fashion via mask-based WPD are investigated. The error cost mismatch between the speech enhancement front-end and ASR back-end components is minimized by end-to-end jointly fine-tuning using either the ASR cost function alone, or its interpolation with the speech enhancement loss. Experiments were conducted on the mixture overlapped and reverberant speech data constructed using simulation or replay of the Oxford LRS2 dataset. The proposed audio-visual multi-channel speech separation, dereverberation and recognition systems consistently outperformed the comparable audio-only baseline by 9.1% and 6.2% absolute (41.7% and 36.0% relative) word error rate (WER) reductions. Consistent speech enhancement improvements were also obtained on PESQ, STOI and SRMR scores 1 .

Audio-Visual Source Separation Based Fusion Techniques

“…Li et al [22] created a novel audio-visual deep learning technique that combines auditory and visual data to detect speech from many channels. The separation filters that extract the desired speech from a mixed input of microphones and video frames are constructed by a neural network.…”

“…Li et al [22] Estimates target speech separation filters from multiple microphones and video frames. A multi-task framework addresses dereverberation and voice recognition tasks.…”

An Overview of Audio-Visual Source Separation Using Deep Learning

“…Li et al [14] Proposed an AV deep learning approach for multi-channel speech separation by jointly modeling audio-visual cues. It includes a neural network that estimates separation filters for target speech from multiple microphones and video frames, and a multi-task framework for dereverberation and speech recognition.…”

“…One advanced method for audio-visual source separation involves the use of deep learning techniques [10][11][12]14,22,34,[37][38].…”

Identity of university Chinese heritage language learners in Hong Kong

“…In recent years, end-to-end DNN-based microphone array beamforming techniques represented by a) neural timefrequency (TF) masking approaches [7]; b) neural Filter and Sum methods [8,9]; and c) mask-based MVDR [10] and generalized eigenvalues (GEV) [11] approaches have been widely adopted. In addition, incorporating visual information into either multi speech separation front-ends alone [12], or further into speech recognition back-ends [13], can further improve the overall system performance.…”