Improving Rare Word Recognition with LM-aware MWER Training

Abstract

Language models (LMs) significantly improve the recognition accuracy of end-to-end (E2E) models on words rarely seen during training, when used in either the shallow fusion or the rescoring setups. In this work, we introduce LMs in the learning of hybrid autoregressive transducer (HAT) models in the discriminative training framework, to mitigate the training versus inference gap regarding the use of LMs. For the shallow fusion setup, we use LMs during both hypotheses generation and loss computation, and the LM-aware MWER-trained model achieves 10% relative improvement over the model trained with standard MWER on voice search test sets containing rare words. For the rescoring setup, we learn a small neural module to generate per-token fusion weights in a data-dependent manner. This model achieves the same rescoring WER as regular MWER-trained model, but without the need for sweeping fusion weights.

Personalization for BERT-based Discriminative Speech Recognition Rescoring

“…In this study, the feed-forward-network f is a linear layer without any activation functions. The rescoring model is trained using a standard minimum word error rate (MWER) loss calculated on n-best hypotheses from the first-pass: loss [28,29,30]:…”

“…The hyperparameters for the model architecture are outlined in Table 2. Both BERT models were first pre-trained on the MC4 dataset [30], and then domain-adapted using internal in-domain data. The BERT models were used to initialize the baseline RescoreBERT models, followed by MWER-based training on the general training data [10].…”