Two-Step Joint Optimization with Auxiliary Loss Function for Noise-Robust Speech Recognition

Abstract

In this paper, a new two-step joint optimization approach based on the asynchronous subregion optimization method is proposed for training a pipeline model composed of two different models. The first-step processing of the proposed joint optimization approach trains the front-end model only, and the second-step processing trains all the parameters of the combined model together. In the asynchronous subregion optimization method, the first-step processing only supports the goal of the front-end model. However, the first-step processing of the proposed approach works with a new loss function to make the front-end model support the goal of the back-end model. The proposed optimization approach was applied, here, to a pipeline composed of a deep complex convolutional recurrent network (DCCRN)-based speech enhancement model and a conformer-transducer-based ASR model as a front-end and a back-end, respectively. Then, the performance of the proposed two-step joint optimization approach was evaluated on the LibriSpeech automatic speech recognition (ASR) corpus in noisy environments by measuring the character error rate (CER) and word error rate (WER). In addition, an ablation study was carried out to examine the effectiveness of the proposed optimization approach on each of the processing blocks in the conformer-transducer ASR model. Consequently, it was shown from the ablation study that the conformer-transducer-based ASR model with the joint network trained only by the proposed optimization approach achieved the lowest average CER and WER. Moreover, the proposed optimization approach reduced the average CER and WER on the Test-Noisy dataset under matched noise conditions by 0.30% and 0.48%, respectively, compared to the approach of separate optimization of speech enhancement and ASR. Compared to the conventional two-step joint optimization approach, the proposed optimization approach provided average CER and WER reductions of 0.22% and 0.31%, respectively. Moreover, it was revealed that the proposed optimization approach achieved a lower average CER and WER, by 0.32% and 0.43%, respectively, than the conventional optimization approach under mismatched noise conditions.

Cluster-Based Pairwise Contrastive Loss for Noise-Robust Speech Recognition

“…The ASR performance of each of the training approaches was evaluated by measuring the WER on both the validation and the test datasets. The WER of the ASR model trained using the proposed training approach was then compared to those of six different approaches, as follows: (1) an ASR model trained via the MCT using the clean and noisy training datasets (denoted as MCT-noisy); (2) an SE model trained on the clean and noisy speech training datasets, wherein the enhanced signal was subsequently fed into the MCT-noisy ASR model (denoted as MCT-noisy + standalone-SE); (3) an ASR model trained by the MCT using the clean, noisy, and enhanced data from the standalone-SE datasets (denoted as MCT-all); (4) a combination of the SE and ASR models trained by conventional joint optimization (denoted as Joint-Straight) [ 9 ]; (5) a pipeline trained by ASO-based joint optimization (denoted as Joint-ASO) [ 18 ]; (6) a pipeline trained by Grad-based joint optimization (denoted as Joint-Grad) [ 20 ]; and (7) a pipeline trained by Token-based joint optimization (denoted as Joint-Token) [ 22 ].…”

“…In this section, the performance of the proposed training approach was evaluated for noise-robust ASR, and it was then compared with the performance of MCT and conventional joint training approaches, including asynchronous subregion optimization (ASO)-based joint optimization [ 18 ], gradients-surgery (Grad)-based joint optimization [ 19 ], and acoustic tokenizer (Token)-based joint optimization [ 22 ]. Here, the ASO-based joint optimization approach was first used to train a pipeline with the SE and ASR encoder losses, and it was then further trained with the combination of the SE and ASR losses.…”

“…The conflicting gradients originate from the different gradient scales and directions between the SE and ASR models, which is caused by the differences in their neural architectures and loss functions, with different task goals. To solve the conflicting problem, several training approaches have been studied, including those based on asynchronous subregion optimization (ASO) [ 18 , 19 ], gradient surgery [ 20 , 21 ], and knowledge distillation (KD) [ 22 , 23 , 24 ].…”

Knowledge Distillation-Based Training of Speech Enhancement for Noise-Robust Automatic Speech Recognition

“…In this section, we evaluate the performance of the proposed KD-based training approach for noise-robust ASR and compare it with the MCT and conventional joint training approaches, including ASO-based joint optimization [25] and gradient-remedy-based joint optimization [24]. The ASR and SE performances were measured using two different datasets.…”

“…When optimizing the entire pipeline, challenges arise because of conflicting gradients, leading to a convergence issue, which is called a conflicting problem [23], [24]. To solve this conflicting problem, several training approaches have been studied, such as the asynchronous subregion optimization (ASO)-based approach [25], [26] and the gradient surgery-based approach [24], [27]. Although these approaches yield promising results, they suffer from frame mismatching between SE and ASR [28], which mainly stems from the different objectives of SE and ASR.…”