Speech-XLNet: Unsupervised Acoustic Model Pretraining for Self-Attention Networks

Abstract

Self-attention network (SAN) can benefit significantly from the bi-directional representation learning through unsupervised pretraining paradigms such as BERT and XLNet. In this paper, we present an XLNet-like pretraining scheme "Speech-XLNet" for unsupervised acoustic model pretraining to learn speech representations with SAN. The pretrained SAN is finetuned under the hybrid SAN/HMM framework. We conjecture that by shuffling the speech frame orders, the permutation in Speech-XLNet serves as a strong regularizer to encourage the SAN to make inferences by focusing on global structures through its attention weights. In addition, Speech-XLNet also allows the model to explore the bi-directional contexts for effective speech representation learning. Experiments on TIMIT and WSJ demonstrate that Speech-XLNet greatly improves the SAN/HMM performance in terms of both convergence speed and recognition accuracy compared to the one trained from randomly initialized weights. Our best systems achieve a relative improvement of 11.9% and 8.3% on the TIMIT and WSJ tasks respectively. In particular, the best system achieves a phone error rate (PER) of 13.3% on the TIMIT test set, which to our best knowledge, is the lowest PER obtained from a single system.

SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding

“…Self-supervised pre-training for speech In speech, wav2vec (Schneider et al, 2019) leverages contrastive learning to produce contextual representations for audio input; vq-wav2vec (Baevski et al, 2020a) and wav2vec 2.0 (Baevski et al, 2020b) further propose to discretize the original continuous audio signals in order to enable more efficient MLM training with Transformer (Vaswani et al, 2017). Pre-trained speech models have been applied to ASR Baevski et al, 2020b), phoneme recognition (Song et al, 2020;Liu et al, 2020a), speech translation (Nguyen et al, 2020;Chung et al, 2019c), and speech synthesis (Chung et al, 2019b), to name a few.…”

Probing Acoustic Representations for Phonetic Properties

“…More recent work has explored incorporating contextual information in the pre-training stage, and model the use of frames in context of the entire input sequence. The pre-training objectives, usually using self-supervised learning, include next step prediction [7,8], masked acoustic modeling [9,10,11], and connectionist temporal classification [12]. Pre-trained contextualized acoustic representations appear to be extremely effective.…”

A Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition

“…Reference [10,11] proposed an Autoregressive Predictive Coding (APC) objective that predicts unseen future frames based on past frames, which yielded satisfactory results in phonetic classification, speech recognition, and speech translation. Other studies [12,13,14,15,16,17] were motivated by NLP and applied similar methods on speech tasks. Among these methods, Masked Predictive Coding (MPC) [15] realized significant improvements on state-of-the-art transformerbased speech recognition models on various datasets without introducing any additional parameters into the speech recognition model.…”