Speech Enhancement Using End-to-End Speech Recognition Objectives

Subramanian, Arvind; Wang, Xiaofei; Baskar, Murali Karthick; Watanabe, Shinji; Taniguchi, Tadatsugu; Trung, Dung Tran; Fujita, Yuya

doi:10.1109/waspaa.2019.8937250

Cited by 50 publications

(17 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…And in this paper, we mainly focus on the multi-speaker case, which is a more difficult task. It is worth noting that this end-to-end architecture is optimized only based on the final ASR criterion, which was also proven feasible in previous works [16,[24][25][26]. Our experiments show that our newly proposed method outperformed the conventional end-to-end ASR systems [24,25,27] in both single-speaker and multi-speaker reverberant conditions.…”

Section: Introductionsupporting

confidence: 52%

End-to-End Far-Field Speech Recognition with Unified Dereverberation and Beamforming

Zhang¹,

Subramanian²,

Chang³

et al. 2020

Interspeech 2020

Self Cite

View full text Add to dashboard Cite

Despite successful applications of end-to-end approaches in multi-channel speech recognition, the performance still degrades severely when the speech is corrupted by reverberation. In this paper, we integrate the dereverberation module into the end-to-end multi-channel speech recognition system and explore two different frontend architectures. First, a multisource mask-based weighted prediction error (WPE) module is incorporated in the frontend for dereverberation. Second, another novel frontend architecture is proposed, which extends the weighted power minimization distortionless response (WPD) convolutional beamformer to perform simultaneous separation and dereverberation. We derive a new formulation from the original WPD, which can handle multi-source input, and replace eigenvalue decomposition with the matrix inverse operation to make the back-propagation algorithm more stable. The above two architectures are optimized in a fully end-toend manner, only using the speech recognition criterion. Experiments on both spatialized wsj1-2mix corpus and REVERB show that our proposed model outperformed the conventional methods in reverberant scenarios.

show abstract

Section: Introductionsupporting

confidence: 52%

End-to-End Far-Field Speech Recognition with Unified Dereverberation and Beamforming

Zhang¹,

Subramanian²,

Chang³

et al. 2020

Interspeech 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…ASR Multi-task Training: There are prior studies that employ joint or multi-task training for the front-end and backend [12,13,14,15,16,17,18,19,20,21,22]. However, they concerned only about the ASR accuracy and did not pay close attention to the SE quality nor analyzed the trade-off between the two tasks.…”

Section: Related Workmentioning

confidence: 99%

Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement

Eskimez

Wang

Tang³

et al. 2021

Interspeech 2021

Self Cite

View full text Add to dashboard Cite

With the surge of online meetings, it has become more critical than ever to provide high-quality speech audio and live captioning under various noise conditions. However, most monaural speech enhancement (SE) models introduce processing artifacts and thus degrade the performance of downstream tasks, including automatic speech recognition (ASR). This paper proposes a multi-task training framework to make the SE models unharmful to ASR. Because most ASR training samples do not have corresponding clean signal references, we alternately perform two model update steps called SE-step and ASR-step. The SEstep uses clean and noisy signal pairs and a signal-based loss function. The ASR-step applies a pre-trained ASR model to training signals enhanced with the SE model. A cross-entropy loss between the ASR output and reference transcriptions is calculated to update the SE model parameters. Experimental results with realistic large-scale settings using ASR models trained on 75,000-hour data show that the proposed framework improves the word error rate for the SE output by 11.82% with little compromise in the SE quality. Performance analysis is also carried out by changing the ASR model, the data used for the ASR-step, and the schedule of the two update steps.

show abstract

“…With the popularity of speech-related intelligent devices and related applications, front-end processing has become a popular research topic [1]. Among them, a batch of methods based on end-to-end deep learning have emerged to solve the cocktail party problem [2]- [7]. Compared to conventional approaches like computational auditory scene analysis [8], [9] and non-negative matrix factorization [10], end-to-end models are entirely data-driven, achieving remarkable improvement in speech quality and intelligibility.…”

Section: Introductionmentioning

confidence: 99%

Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem

Shi¹,

Chang²,

Tomoki³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Deep learning based models have significantly improved the performance of speech separation with input mixtures like the cocktail party. Prominent methods (e.g., frequency-domain and time-domain speech separation) usually build regression models to predict the ground-truth speech from the mixture, using the masking-based design and the signal-level loss criterion (e.g., MSE or SI-SNR). This study demonstrates, for the first time, that the synthesis-based approach can also perform well on this problem, with great flexibility and strong potential. Specifically, we propose a novel speech separation/enhancement model based on the recognition of discrete symbols, and convert the paradigm of the speech separation/enhancement related tasks from regression to classification. By utilizing the synthesis model with the input of discrete symbols, after the prediction of discrete symbol sequence, each target speech could be resynthesized. Evaluation results based on the WSJ0-2mix and VCTKnoisy corpora in various settings show that our proposed method can steadily synthesize the separated speech with high speech quality and without any interference, which is difficult to avoid in regressionbased methods. In addition, with negligible loss of listening quality, the speaker conversion of enhanced/separated speech could be easily realized through our method.

show abstract

Speech Enhancement Using End-to-End Speech Recognition Objectives

Cited by 50 publications

References 20 publications

End-to-End Far-Field Speech Recognition with Unified Dereverberation and Beamforming

End-to-End Far-Field Speech Recognition with Unified Dereverberation and Beamforming

Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement

Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem

Contact Info

Product

Resources

About