Semi-Supervised Speech Recognition Via Graph-Based Temporal Classification

Abstract

Semi-supervised learning has demonstrated promising results in automatic speech recognition (ASR) by self-training using a seed ASR model with pseudo-labels generated for unlabeled data. The effectiveness of this approach largely relies on the pseudo-label accuracy, for which typically only the 1-best ASR hypothesis is used. However, alternative ASR hypotheses of an N -best list can provide more accurate labels for an unlabeled speech utterance and also reflect uncertainties of the seed ASR model. In this paper, we propose a generalized form of the connectionist temporal classification (CTC) objective that accepts a graph representation of the training labels. The newly proposed graph-based temporal classification (GTC) objective is applied for self-training with WFST-based supervision, which is generated from an N -best list of pseudo-labels. In this setup, GTC is used to learn not only a temporal alignment, similarly to CTC, but also a label alignment to obtain the optimal pseudo-label sequence from the weighted graph. Results show that this approach can effectively exploit an N -best list of pseudo-labels with associated scores, considerably outperforming standard pseudo-labeling, with ASR results approaching an oracle experiment in which the best hypotheses of the N -best lists are selected manually.

Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition

“…Moreover, MPL was shown to be effective independently of the amount of unlabeled data or domain mismatch. Future work includes applying filtering techniques [31] and introducing multiple hypotheses [32] in the MPL process.…”

“…We focus on self-training [21] or pseudo-labeling (PL) [22], which has recently been adopted for semi-supervised E2E ASR and shown to be effective [23][24][25][26][27][28][29][30][31][32]. In PL, a teacher (base) model is first trained on labeled data and used to generate * Research conducted during an internship at MERL pseudo-labels for unlabeled data.…”

Extended Graph Temporal Classification for Multi-Speaker End-to-End ASR

“…In this section, we describe the extended GTC loss function. For the convenience of understanding, we mostly follow the notations in the previous GTC study [14].…”

“…As in GTC [14], we can apply a beam search algorithm during decoding. Since the output of GTC-e contains tokens from multiple speakers, we need to make modifications to the existing timesynchronous prefix beam search algorithm [25,26].…”

“…Recently, researchers proposed to use graph representations in the loss function for training deep neural networks [13]. In [14], a new loss function, called graph-based temporal classification (GTC), was proposed as a generalization of CTC to handle sequence-to-sequence problems. GTC can take graph-based supervisory information as an input to describe all possible alignments between an input sequence and an output sequence, for learning the best possible alignment from the training data.…”