Sequence-to-Sequence Automatic Speech Recognition with Word Embedding Regularization and Fused Decoding

Abstract

In this paper, we investigate the benefit that off-the-shelf word embedding can bring to the sequence-to-sequence (seqto-seq) automatic speech recognition (ASR). We first introduced the word embedding regularization by maximizing the cosine similarity between a transformed decoder feature and the target word embedding. Based on the regularized decoder, we further proposed the fused decoding mechanism. This allows the decoder to consider the semantic consistency during decoding by absorbing the information carried by the transformed decoder feature, which is learned to be close to the target word embedding. Initial results on LibriSpeech demonstrated that pre-trained word embedding can significantly lower ASR recognition error with a negligible cost, and the choice of word embedding algorithms among Skip-gram, CBOW and BERT is important.

Automatic Speech Recognition: Systematic Literature Review

“…Other recent studies have sought to improve E2E ASR with word embedding learned from text-only data. In [43], the researchers chose to adopt word embedding because off-the-shelf word embedding carrying semantic information learned from a vast amount of text can be easily obtained. An autoregressive decoder was generally used to predict the transcription corresponding to the input speech.…”

Speech-Based Depression Prediction Using Encoder-Weight-Only Transfer Learning and a Large Corpus

“…The ASR sub-model is based on a hybrid connectionist temporal classification (CTC)/attention architecture [31] and is inspired by prior work including that in [31], [34], and [35]. To train the ASR task, we used the Librispeech dataset [36], which is an English dataset comprising over 1000 hours of read speech.…”