A Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition

Abstract

The construction of an effective good speech recognition system typically requires large amounts of transcribed data, which is expensive to collect. To overcome this problem, many unsupervised pretraining methods have been proposed. Among these methods, Masked Predictive Coding achieved significant improvements on various speech recognition datasets with BERT-like Masked Reconstruction loss and transformer backbone. However, many aspects of MPC have yet to be fully investigated. In this paper, we conduct a further study on MPC and focus on three important aspects: the effect of pretraining data speaking style, its extension on streaming model, and strategies for better transferring learned knowledge from pretraining stage to downstream tasks. The experimental results demonstrated that pretraining data with a matching speaking style is more useful on downstream recognition tasks. A unified training objective with APC and MPC provided an 8.46% relative error reduction on the streaming model trained on HKUST. Additionally, the combination of target data adaption and layerwise discriminative training facilitated the knowledge transfer of MPC, which realized 3.99% relative error reduction on AISHELL over a strong baseline.

BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data

“…Unfortunately, the highly correlated nature of neighbouring samples in EEG (or most other continuous data for that matter), is not conducive to this approach. The likely result would be that, instead of an EM, a method for interpolation would be learned, as has been argued in similar work in self-supervised learning with speech [41]. In other words, the smoothness of these data would make it hard to produce general features simply through recovering missing points.…”

“…Prior work in self-supervised speech recognition has begun to synthesize parts of CPC and MLM to produce methodologies for self-learning with raw waveforms [13,44,45,41,31]. In our work, we adapt one of these approaches called wav2vec 2.0 [13] (its particular formulation is detailed in section 2.4.1) to EEG, and investigate how effective the representations (BENDR) are for downstream tasks.…”

Didispeech: A Large Scale Mandarin Speech Corpus

“…For ASR, several open-source read speech corpora, such as HKUST and AISHELL-1, have also been released. The scale of these databases is significantly smaller than the ASR datasets arXiv:2010.09275v3 [eess.AS] 23 Oct 2020 used in companies [24,25], which may lead to a divergence between research and industry.…”

“…We use characters as labels to train and evaluate all ASR models. Recently, Transformer-based models have achieved competitive performance on HKUST, AISHELL-1 and Lib-riSpeech [24,25,31]. In the ASR experiments, we follow model structure of previous work [25] with e = 12, d = 6, d model = 512, d f f = 1280 and d head = 8.…”

Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data

“…Self-supervised learning has been shown effective in natural language processing (NLP), e.g., BERT [1] and BART [2], where it makes use of a large amount of unlabeled data for pretraining to improve the performance of downstream tasks, such as automatic speech recognition (ASR) [3,4,5,6,7,8].…”