Lip-Reading Driven Deep Learning Approach for Speech Enhancement

Abstract

This paper proposes a novel lip-reading driven deep learning framework for speech enhancement. The proposed approach leverages the complementary strengths of both deep learning and analytical acoustic modelling (filtering based approach) as compared to recently published, comparatively simpler benchmark approaches that rely only on deep learning. The proposed audio-visual (AV) speech enhancement framework operates at two levels. In the first level, a novel deep learning based lipreading regression model is employed. In the second level, lipreading approximated clean-audio features are exploited, using an enhanced, visually-derived Wiener filter (EVWF), for the clean audio power spectrum estimation. Specifically, a stacked longshort-term memory (LSTM) based lip-reading regression model is designed for clean audio features estimation using only temporal visual features (i.e. lip reading) considering different number of prior visual frames. For clean speech spectrum estimation, a new filterbank-domain EVWF is formulated, which exploits estimated speech features. The proposed EVWF is compared with conventional Spectral Subtraction (SS) and Log-Minimum Mean-Square Error (LMMSE) methods using both ideal AV mapping and LSTM driven AV mapping. The potential of the proposed speech enhancement framework is evaluated under four different dynamic real-world commercially-motivated scenarios (cafe, street junction, public transport (BUS), pedestrian area) at different SNR levels (ranging from low to high SNRs) using benchmark Grid and ChiME3 corpora. For objective testing, perceptual evaluation of speech quality (PESQ) is used to evaluate the quality of restored speech. For subjective testing, the standard mean-opinion-score (MOS) method is used with inferential statistics. Comparative simulation results demonstrate significant lip-reading and speech enhancement improvement in terms of both speech quality and speech intelligibility. Ongoing work is aimed at enhancing the accuracy and generalization capability of the deep learning driven lip-reading model, and contextual integration of AV cues, for context-aware autonomous AV speech enhancement.

Deep-learning-based audio-visual speech enhancement in presence of Lombard effect

“…Nevertheless, the datasets collected in controlled environments are a good choice for training a prototype designed for a specific purpose or for studying a particular problem. Examples of databases useful in this sense are: TCD-TIMIT [89] and OuluVS2 [16], to study the influence of several angles of view; MODALITY [46] and OuluVS2 [16], to determine the effect of different video frame rates; Lombard GRID [13], to understand the impact of the Lombard effect, also from several angles of view; RAVDESS [161], to perform a study of emotions in the context of SE and SS; KinectDigits [224] and MODALITY [46], to determine [3], [5], [6], [17], [65], [66], [76], [77] (1,000 of 3 seconds 25 FPS Frontal face [108], [136], [154], [164], [165], [263] per speaker) [176], [183], [195], [203], [239] OuluVS [286] 20 ( singing at two levels of intensity VoxCeleb2 [38] 6,112 Continuous sentences Not specified b Not specified b Videos in the wild from [7], [42], [122], [128], [153], [ Challenging visual and auditory environments LRS2 [8] Hundreds Continuous sentences Not specified b Not specified b Videos in the wild [7], [10], [107], [153]…”

“…Magnitude spectrogram [3]- [7], [10], [12], [17], [37], [42], [65], [66], [76], [77], [85], [99], [100], [107], [122], [128], [136], [153], [154], [164] [165], [176], [178], [179], [183], [192], [195], [203], [208], [220]- [222], [244], [263], [274], [279] Phase a [7], [10], [153] Complex spectrogram [55], [107], [109], [169], [239] Raw waveform [108], [273] Speaker embeddings [10], [85], [169], [192], [ [6],…”

CochleaNet: A robust language-independent audio-visual model for real-time speech enhancement

“…In the literature, extensive research has been carried out to develop audioonly (A-only) and audio-visual (AV) SE methods. Researchers have proposed several SE models such as deep neural network (DNN) based spectral mask estimation models [12,13], DNN based clean spectrogram estimation models [14,15], Wiener filtering based hybrid models [16,17,18], and time-domain SE models [19,20,21]. However, limited work has been conducted to develop robust language-independent, causal, speaker and noise-independent AV SE models for low SNRs (< −3 dB) observed in everyday social environments (such as cafeteria, and restaurants) where traditional A-only hearing aids fail to improve 3 the speech intelligibility.…”

“…However, the model fails to work when the visuals are occluded. Adeel et al[17,18] proposed a visual-only and AV SE models by integrating an enhanced visually-derived wiener filter (EVWF) and DNN based lip reading regression model. The preliminary evaluation demonstrated the effectiveness to deal with spectro-temporal variations in any wide variety of noisy environments.…”

The Role of Multisensory Temporal Covariation in Audiovisual Speech Recognition in Noise

“…The AVIndex provided remarkably good predictions of overall audiovisual-speech intelligibility given its simplicity, suggesting that tweaks designed to address the model's faults as noted in Experiments 1 and 2 (e.g., over-prediction of intelligibility at very low SNRs) could yield a very accurate and informative model. Recent machine learning models designed to perform audiovisual speech enhancement suggest that it is possible to derive local, frame-byframe estimates of acoustic speech features directly from short segments of concurrent visual speech Adeel, Gogate, Hussain, & Whitmer, 2018).…”