Improved Lite Audio-Visual Speech Enhancement

Chuang, Shang-Yi; Wang, Hsin-min; Tsao, Yu

doi:10.1109/taslp.2022.3153265

Cited by 17 publications

(6 citation statements)

References 93 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In this section, we compare the experimental results of the proposed AVSE model with those of other baseline SE models, including two AVSE models [4,23] and one traditional audio-only methods, namely LogMMSE [24]. We conducted an objective comparison with two standardized evaluation metrics that are widely used to evaluate SE performance-the perceptual evaluation of speech quality (PESQ) [25] and short-time objective intelligibility measure (STOI) [26].…”

Section: Resultsmentioning

confidence: 99%

“…For the audio components, we followed the setup described in [4]. To form clean-noisy speech pairs for training, the utterances were artificially corrupted by 100 types of noise [20] at five different signal-to-noise ratios (SNRs) ranging from −12 to 12 dB with an increment of 6 dB.…”

Section: Methodsmentioning

confidence: 99%

“…Mod. PESQ STOI Noisy 1.10 0.76 LogMMSE [24] (A) 1.16 0.77 AV-CVAE [23] (AV) 1.26 0.79 LAVSE [4] (AV)…”

Section: Methodsunclassified

“…Speech enhancement (SE) and speech separation (SS) aim to extract speech signals of interest from a given utterance mixed with unwanted audio signals. With recent developments in deep learning (DL), DL-based methods have demonstrated better results than traditional SE and SS methods, either for audio-only or audio-visual (AV) applications [1,2,3,4,5,6]. Nevertheless, most DL-based AVSE and AVSS models have their own specific modules designed to better integrate the audio-visual information for the target task, which may not be favorable from the viewpoints of some current DL model design philosophies.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Audio-Visual Speech Enhancement and Separation by Leveraging Multi-Modal Self-Supervised Embeddings

Chern¹,

Hung²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

AV-HuBERT, a multi-modal self-supervised learning model, has been shown to be effective for categorical problems such as automatic speech recognition and lip-reading. This suggests that useful audio-visual speech representations can be obtained via utilizing multi-modal self-supervised embeddings. Nevertheless, it is unclear if such representations can be generalized to solve real-world multi-modal AV regression tasks, such as audio-visual speech enhancement (AVSE) and audio-visual speech separation (AVSS). In this study, we leveraged the pre-trained AV-HuBERT model followed by an SE module for AVSE and AVSS. Comparative experimental results demonstrate that our proposed model performs better than the state-of-the-art AVSE and traditional audio-only SE models. In summary, our results confirm the effectiveness of our proposed model for the AVSS task with proper fine-tuning strategies, demonstrating that multi-modal selfsupervised embeddings obtained from AV-HUBERT can be generalized to audio-visual regression tasks.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

“…Mod. PESQ STOI Noisy 1.10 0.76 LogMMSE [24] (A) 1.16 0.77 AV-CVAE [23] (AV) 1.26 0.79 LAVSE [4] (AV)…”

Section: Methodsunclassified

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Audio-Visual Speech Enhancement and Separation by Leveraging Multi-Modal Self-Supervised Embeddings

Chern¹,

Hung²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Chuang, S.Y., et al [12] proposed an improved lite audio-visual speech enhancement (iLAVSE) algorithm for a car-driving scenario. Three stages are involved in the iLAVSE system: data preprocessing, AVSE based on CRNN, and reconstruction.…”

Section: Literature Surveymentioning

confidence: 99%

Microphone Array Speech Enhancement Via Beamforming Based Deep Learning Network

Pathrose,

Ismail M,

Mohan P

2023

Int. j. electr. comput. eng. syst. (Online)

View full text Add to dashboard Cite

In general, in-car speech enhancement is an application of the microphone array speech enhancement in particular acoustic environments. Speech enhancement inside the moving cars is always an interesting topic and the researchers work to create some modules to increase the quality of speech and intelligibility of speech in cars. The passenger dialogue inside the car, the sound of other equipment, and a wide range of interference effects are major challenges in the task of speech separation in-car environment. To overcome this issue, a novel Beamforming based Deep learning Network (Bf-DLN) has been proposed for speech enhancement. Initially, the captured microphone array signals are pre-processed using an Adaptive beamforming technique named Least Constrained Minimum Variance (LCMV). Consequently, the proposed method uses a time-frequency representation to transform the pre-processed data into an image. The smoothed pseudo-Wigner-Ville distribution (SPWVD) is used for converting time-domain speech inputs into images. Convolutional deep belief network (CDBN) is used to extract the most pertinent features from these transformed images. Enhanced Elephant Heard Algorithm (EEHA) is used for selecting the desired source by eliminating the interference source. The experimental result demonstrates the effectiveness of the proposed strategy in removing background noise from the original speech signal. The proposed strategy outperforms existing methods in terms of PESQ, STOI, SSNRI, and SNR. The PESQ of the proposed Bf-DLN has a maximum PESQ of 1.98, whereas existing models like Two-stage Bi-LSTM has 1.82, DNN-C has 1.75 and GCN has 1.68 respectively. The PESQ of the proposed method is 1.75%, 3.15%, and 4.22% better than the existing GCN, DNN-C, and Bi-LSTM techniques. The efficacy of the proposed method is then validated by experiments.

show abstract