End-to-end Contextual Speech Recognition Using Class Language Models and a Token Passing Decoder

Chen, Zhehuai; Jain, Mahaveer; Wang, Yongqiang; Seltzer, Michael L.; Fuegen, Christian

doi:10.1109/icassp.2019.8683573

Cited by 39 publications

(41 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This biasing can be applied at word boundaries [8], at the grapheme level [11,13,10], or at the subword level [13,14]. Given that E2E models generally used a constrained beam [16], applying biasing only at word boundaries cannot improve performance if the relevant word does not already appear in the beam.…”

Section: Previous Workmentioning

confidence: 99%

Personalization Strategies for End-to-End Speech Recognition Systems

Gourav

Liu

Gandhe

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

The recognition of personalized content, such as contact names, remains a challenging problem for end-to-end speech recognition systems. In this work, we demonstrate how first-and second-pass rescoring strategies can be leveraged together to improve the recognition of such words. Following previous work, we use a shallow fusion approach to bias towards recognition of personalized content in the first-pass decoding. We show that such an approach can improve personalized content recognition by up to 16% with minimum degradation on the general use case. We describe a fast and scalable algorithm that enables our biasing models to remain at the wordlevel, while applying the biasing at the subword level. This has the advantage of not requiring the biasing models to be dependent on any subword symbol table. We also describe a novel second-pass de-biasing approach: used in conjunction with a first-pass shallow fusion that optimizes on oracle WER, we can achieve an additional 14% improvement on personalized content recognition, and even improve accuracy for the general use case by up to 2.5%.

show abstract

Section: Previous Workmentioning

confidence: 99%

Personalization Strategies for End-to-End Speech Recognition Systems

Gourav

Liu

Gandhe

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Model The objective of this task is to generate reasonable sentences from fed-in audio utterances.The baseline model proposed for this task adopts an encoder-decoder approach, similar to that of endto-end automatic speech recognition [19,20] and image captioning [6,7] tasks. The encoder outputs a single fixed dimensional vector u for each utterance.…”

Section: Featurename Window Shift Dimensionmentioning

confidence: 99%

Audio Caption: Listen and Tell

Dinkel

2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Increasing amount of research has shed light on machine perception of audio events, most of which concerns detection and classification tasks. However, human-like perception of audio scenes involves not only detecting and classifying audio sounds, but also summarizing the relationship between different audio events. Comparable research such as image caption has been conducted, yet the audio field is still quite barren. This paper introduces a manually-annotated dataset for audio caption. The purpose is to automatically generate natural sentences for audio scene description and to bridge the gap between machine perception of audio and image. The whole dataset is labelled in Mandarin and we also include translated English annotations. A baseline encoder-decoder model is provided for both English and Mandarin. Similar BLEU scores are derived for both languages: our model can generate understandable and data-related captions based on the dataset.

show abstract

“…Shallow fusion [10,9] solves this by generating on-the-fly contextual LMs that are interpolated with E2E neural model's scores to bias the beam search during decoding. This method was further improved by using a token-passing decoder with efficient token recombination to minimize search errors when the number of contextual entities is large [15]. While this showed improvements over standard shallow fusion, there are still gaps between E2E and traditional modular systems in contextual ASR.…”

Section: Contextual Speech Recognition Using Shallow Fusionmentioning

confidence: 99%

“…The main network follows the LAS architecture [12], with a 2-layer BLSTM with 1400 hidden nodes per layer as the encoder and 2-layer LSTM with 700 hidden nodes as the decoder. More details can be found in [15]. We built the system with PyTorch [25] based on Espnet [26], and implemented Block-momentum SGD [27] to enable distributed training with linear speedups and no performance degradation.…”

Section: Setupmentioning

confidence: 99%

Joint Grapheme and Phoneme Embeddings for Contextual End-to-End ASR

et al. 2019

Self Cite

View full text Add to dashboard Cite

End-to-end approaches to automatic speech recognition, such as Listen-Attend-Spell (LAS), blend all components of a traditional speech recognizer into a unified model. Although this simplifies training and decoding pipelines, a unified model is hard to adapt when mismatch exists between training and test data, especially if this information is dynamically changing. The Contextual LAS (CLAS) framework tries to solve this problem by encoding contextual entities into fixed-dimensional embeddings and utilizing an attention mechanism to model the probabilities of seeing these entities. In this work, we improve the CLAS approach by proposing several new strategies to extract embeddings for the contextual entities. We compare these embedding extractors based on graphemic and phonetic input and/or output sequences and show that an encoder-decoder model trained jointly towards graphemes and phonemes outperforms other approaches. Leveraging phonetic information obtains better discrimination for similarly written graphemic sequences and also helps the model generalize better to graphemic sequences unseen in training. We show significant improvements over the original CLAS approach and also demonstrate that the proposed method scales much better to a large number of contextual entities across multiple domains.

show abstract

End-to-end Contextual Speech Recognition Using Class Language Models and a Token Passing Decoder

Cited by 39 publications

References 28 publications

Personalization Strategies for End-to-End Speech Recognition Systems

Personalization Strategies for End-to-End Speech Recognition Systems

Audio Caption: Listen and Tell

Joint Grapheme and Phoneme Embeddings for Contextual End-to-End ASR

Contact Info

Product

Resources

About