Momentum Pseudo-Labeling: Semi-Supervised ASR With Continuously Improving Pseudo-Labels

Higuchi, Yuki; Moritz, Niko; Roux, Jonathan Le; Hori, Takaaki

doi:10.1109/jstsp.2022.3195367

Cited by 15 publications

(9 citation statements)

References 76 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…transcription generated by some method. Different ways of inferring pseudo-labels PL(x; θ ) have been proposed [22,31,38,26,29,18,7], including both greedy and beam-search decoding, with or without an external LM, and with variants on the "teacher" AM model θ . IPL [38] and slimIPL [26] are continuous PL approaches, where a single AM (with parameters θ) is continuously trained.…”

Section: Acoustic (Am) and Language (Lm) Modelsmentioning

confidence: 99%

“…The two dominant methods for leveraging unlabeled audio are unsupervised pre-training via selfsupervision (SSL) [6,19,11,4] and semi-supervised self-training [22,38,26,29,16,18], or pseudo-labeling (PL). In pre-training, a model is trained to process the raw unlabeled data to extract features that solve some pretext task, followed by supervised fine-tuning on some downstream ASR task.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

slimIPL: Language-Model-Free Iterative Pseudo-Labeling

Likhomanenko

Kahn

et al. 2021

Interspeech 2021

View full text Add to dashboard Cite

Recent results in end-to-end ASR have demonstrated the efficacy of simple pseudo-labeling for semisupervised models trained both with Connectionist Temporal Classification (CTC) and Sequenceto-Sequence (seq2seq) losses. Iterative Pseudo-Labeling (IPL), which continuously trains a single model using pseudo-labels iteratively re-generated as the model learns, has been shown to further increase performance in ASR. We improve upon the IPL algorithm: as the model learns, we propose to iteratively re-generate transcriptions with hard labels (the most probable tokens) assignments, that is without a language model. We call this approach Language-Model-Free IPL (slimIPL) and we give a resultant training setup for CTC and seq2seq models. At inference, our experiments show that decoding with a strong language model is more beneficial with slimIPL than IPL, as IPL exhibits some language model over-fitting issues. Compared to prior work on semi-supervised and unsupervised approaches, slimIPL not only simplifies the training process, but also achieves competitive and state-of-the-art results on LIBRISPEECH test sets in both standard and low-resource settings.

show abstract

Section: Acoustic (Am) and Language (Lm) Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

slimIPL: Language-Model-Free Iterative Pseudo-Labeling

Likhomanenko

Kahn

et al. 2021

Interspeech 2021

View full text Add to dashboard Cite

show abstract

“…Another technique is the use of an exponential moving average (EMA) of the acoustic model to generate the pseudo-labels in Eq. ( 2) (Likhomanenko et al, 2021a;Manohar et al, 2021;Higuchi et al, 2021Higuchi et al, , 2022bZhang et al, 2022).…”

Section: Experimental Setup and Related Methodsmentioning

confidence: 99%

“…10k 20k 40k WER 14.3 17.1 22.9 (teacher-student) Kahn et al, 2020a;, here pseudo-labels (PLs) are generated online with a very recent version of the model Likhomanenko et al, 2021a;Manohar et al, 2021;Higuchi et al, 2021Higuchi et al, , 2022a and training is faster and more resource-efficient. One of the main challenges for continuous ST is training stability (Likhomanenko et al, 2021a;Higuchi et al, 2021Higuchi et al, , 2022bCai et al, 2022). While these prior works use various techniques for stabilization, one common ingredient is that models are initially trained on labeled data for M steps.…”

mentioning

confidence: 99%

Continuous Pseudo-Labeling from the Start

Berrebbi¹,

Collobert²,

Bengio³

et al. 2022

Preprint

View full text Add to dashboard Cite

Self-training (ST), or pseudo-labeling has sparked significant interest in the automatic speech recognition (ASR) community recently because of its success in harnessing unlabeled data. Unlike prior semi-supervised learning approaches that relied on iteratively regenerating pseudo-labels (PLs) from a trained model and using them to train a new model, recent state-of-the-art methods perform 'continuous training' where PLs are generated using a very recent version of the model being trained. Nevertheless, these approaches still rely on bootstrapping the ST using an initial supervised learning phase where the model is trained on labeled data alone. We believe this has the potential for over-fitting to the labeled dataset in low resource settings and that ST from the start of training should reduce over-fitting. In this paper we show how we can do this by dynamically controlling the evolution of PLs during the training process in ASR. To the best of our knowledge, this is the first study that shows the feasibility of generating PLs from the very start of the training. We are able to achieve this using two techniques that avoid instabilities which lead to degenerate models that do not generalize. Firstly, we control the evolution of PLs through a curriculum that uses the online changes in PLs to control the membership of the cache of PLs and improve generalization. Secondly, we find that by sampling transcriptions from the predictive distribution, rather than only using the best transcription, we can stabilize training further. With these techniques, our ST models match prior works without an external language model.

show abstract

“…It utilizes labeled samples to predict the class of unlabeled samples and integrates labeled and pseudo-labeled samples to train the network. Semi-supervised learning methods based on the pseudo label have been gradually applied to automatic speech recognition [ 26 ] and image semantic segmentation [ 27 ]. To overcome the limitations of the traditional supervised SAE and to improve the generalization performance, the pseudo label-based semi-supervised stacked autoencoder (PL-SSAE) is proposed by combining the SAE with the pseudo label.…”

Section: Introductionmentioning

confidence: 99%

A Semi-Supervised Stacked Autoencoder Using the Pseudo Label for Classification Tasks

Lai,

Wang,

Xiang

et al. 2023

Entropy

View full text Add to dashboard Cite

The efficiency and cognitive limitations of manual sample labeling result in a large number of unlabeled training samples in practical applications. Making full use of both labeled and unlabeled samples is the key to solving the semi-supervised problem. However, as a supervised algorithm, the stacked autoencoder (SAE) only considers labeled samples and is difficult to apply to semi-supervised problems. Thus, by introducing the pseudo-labeling method into the SAE, a novel pseudo label-based semi-supervised stacked autoencoder (PL-SSAE) is proposed to address the semi-supervised classification tasks. The PL-SSAE first utilizes the unsupervised pre-training on all samples by the autoencoder (AE) to initialize the network parameters. Then, by the iterative fine-tuning of the network parameters based on the labeled samples, the unlabeled samples are identified, and their pseudo labels are generated. Finally, the pseudo-labeled samples are used to construct the regularization term and fine-tune the network parameters to complete the training of the PL-SSAE. Different from the traditional SAE, the PL-SSAE requires all samples in pre-training and the unlabeled samples with pseudo labels in fine-tuning to fully exploit the feature and category information of the unlabeled samples. Empirical evaluations on various benchmark datasets show that the semi-supervised performance of the PL-SSAE is more competitive than that of the SAE, sparse stacked autoencoder (SSAE), semi-supervised stacked autoencoder (Semi-SAE) and semi-supervised stacked autoencoder (Semi-SSAE).

show abstract

Momentum Pseudo-Labeling: Semi-Supervised ASR With Continuously Improving Pseudo-Labels

Cited by 15 publications

References 76 publications

slimIPL: Language-Model-Free Iterative Pseudo-Labeling

slimIPL: Language-Model-Free Iterative Pseudo-Labeling

Continuous Pseudo-Labeling from the Start

A Semi-Supervised Stacked Autoencoder Using the Pseudo Label for Classification Tasks

Contact Info

Product

Resources

About