Bridging the Gap Between Monaural Speech Enhancement and Recognition With Distortion-Independent Acoustic Modeling

Abstract

Monaural speech enhancement has made dramatic advances since the introduction of deep learning a few years ago.Although enhanced speech has been demonstrated to have better intelligibility and quality for human listeners, feeding it directly to automatic speech recognition (ASR) systems trained with noisy speech has not produced expected improvements in ASR performance. The lack of an enhancement benefit on recognition, or the gap between monaural speech enhancement and recognition, is often attributed to speech distortions introduced in the enhancement process. In this study, we analyze the distortion problem, compare different acoustic models, and investigate a distortionindependent training scheme for monaural speech recognition. Experimental results suggest that distortion-independent acoustic modeling is able to overcome the distortion problem. Such an acoustic model can also work with speech enhancement models different from the one used during training. Moreover, the models investigated in this paper outperform the previous best system on the CHiME-2 corpus.

A Deep Learning Speech Enhancement Architecture Optimised for Speech Recognition and Hearing Aids

“…In deep learning-based SE, the noisy speech y is processed by a DNN to generate an estimate to the clean speech signal, ŝ. As proved in [4], [8], the SE process adds some unwanted artifacts that negatively affects the enhanced speech signal, as it causes speech distortion. Considering the effect of these artifacts, the time domain enhanced speech signal, ŝ2 , that is generated by the second stage DE-CADE network, shown in Fig.…”

“…Recent deep learning-based speech enhancement (SE) architectures have shown a great ability to generate estimated clean speech signals with high quality and intelligibility [1]- [3]. This allows these architectures to be employed for real-life SE applications, including Automatic Speech Recognition (ASR) [4], [5] and hearing aids [6], [7]. However, when applying SE architectures to these applications, other factors should be taken into consideration.…”

“…On the one hand, SE is not always useful for applications such as ASR, because the artifacts added by the distortion caused by the enhancement networks sometimes result in worse Word Error Rates (WERs) [4], [8]. To solve this issue, two-stage SE networks can be implemented to minimize distortion [3], [9].…”

“…To solve this issue, two-stage SE networks can be implemented to minimize distortion [3], [9]. Another idea is the joint training of the SE network and ASR system, which also shows better performance [4]. Although these solutions result in improving the WER, the SE network should only be turned on when necessary.…”

Audio-Visual Speech Separation and Dereverberation With a Two-Stage Multimodal Network

“…First, ratio masking is well justified for separation under the assumption that target speech and background interference are uncorrelated, which holds well for additive noise (including background noise and interfering speech) but not for convolutive interference as in the case of reverberation [40]. Second, speech separation algorithms commonly introduce processing artifacts into the target speech signal [12], [42]. It is likely difficult for ratio masking to suppress such processing artifacts introduced by the separation module, particularly considering that these artifacts are correlated with the target speech signal.…”