Sentence-Select: Large-Scale Language Model Data Selection for Rare-Word Speech Recognition

Abstract

Language model fusion helps smart assistants recognize words which are rare in acoustic data but abundant in textonly corpora (typed search logs). However, such corpora have properties that hinder downstream performance, including being (1) too large, (2) beset with domain-mismatched content, and (3) heavy-headed rather than heavy-tailed (excessively many duplicate search queries such as "weather"). We show that three simple strategies for selecting language modeling data can dramatically improve rare-word recognition without harming overall performance. First, to address the heavy-headedness, we downsample the data according to a soft log function, which tunably reduces high frequency (head) sentences. Second, to encourage rare-word exposure, we explicitly filter for words rare in the acoustic data. Finally, we tackle domain-mismatch via perplexity-based contrastive selection, filtering for examples matched to the target domain. We down-select a large corpus of web search queries by a factor of 53x and achieve better LM perplexities than without down-selection. When shallow-fused with a state-of-the-art, production speech engine, our LM achieves WER reductions of up to 24% relative on rare-word sentences (without changing overall WER) compared to a baseline LM trained on the raw corpus. These gains are further validated through favorable side-by-side evaluations on live voice search traffic.

Improving Rare Word Recognition with LM-aware MWER Training

“…For training the LMs, each minibatch is sampled 50/50 from the transcripts of acoustic training data with a total of 150M unique transcripts, and text-only data which contains 50B utterances. The text-only data contains textual search queries from the domains of Maps, Qsession, News, Play, and Youtube, and a frequency-based pruning strategy, designed to improve rare word modeling, is implemented to adjust the probability of selecting each query [25]. All acoustic and LM training data is anonymized and adheres to Google AI Principles [26].…”

A survey on deep reinforcement learning for audio-based applications

“…The results highlight the model's competitive performance across these diverse datasets. In a different study, Huang [169] introduces strategies for curating language modelling data to enhance the recognition of rare words without compromising overall performance. These strategies demonstrate substantial impact, leading to an enhanced language model achieving a noteworthy up to 24% relative reduction in Word Error Rate (WER) for sentences containing rare words.…”

“…8: A prompt-completion example for Launch-PadGPT [262]: The text following "prompt:" represents MFCC feature values, while "completion:" shows RGB-X tuples. The tuple (245,5,169,1) indicates that the Launchpad keyboard's second button (index 0 for the first) is purple.. Figure taken from [262].…”

Unsupervised Data Selection via Discrete Speech Representation for ASR

“…For the data selection module, in order to select data that is close to the target domain, we adopt the language model based contrastive method introduced in [21]. Contrastive data selection is a well-known technique in the NLP and speech communities, and has been widely used in machine translation [22,23], and ASR [24][25][26]. It computes a domain relevance score for each utterance and filters by a threshold to keep those closest to the target domain.…”