An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition

Abstract

Self-supervised pretraining on speech data has achieved a lot of progress. High-fidelity representation of the speech signal is learned from a lot of untranscribed data and shows promising performance. Recently, there are several works focusing on evaluating the quality of self-supervised pretrained representations on various tasks without domain restriction, e.g. SUPERB. However, such evaluations do not provide a comprehensive comparison among many ASR benchmark corpora. In this paper, we focus on the general applications of pretrained speech representations, on advanced end-to-end automatic speech recognition (E2E-ASR) models. We select several pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR. Without any modification of the back-end model architectures or training strategy, some of the experiments with pretrained representations, e.g., WSJ, WSJ0-2mix with HuBERT, reach or outperform current state-of-the-art (SOTA) recognition performance. Moreover, we further explore more scenarios for whether the pretraining representations are effective, such as the cross-language or overlapped speech. The scripts, configuratons and the trained models have been released in ESPnet to let the community reproduce our experiments and improve them.

Self-Supervised Speech Representation Learning: A Review

“…The distributional shifts here may stem from both the acoustics (microphone, room reverberation) as well as lexical effects related to topic and style, as well as differences in speaker characteristics such as accent. Similar problems were also investigated using HuBERT and wav2vec 2.0 models in [280]. In [281] domain effects were studied in greater detail using datasets from six different domains.…”

Self-Supervised Speech Representation Learning: A Review

“…The distributional shifts here may stem from both the acoustics (microphone, room reverberation) as well as lexical effects related to topic and style, as well as differences in speaker characteristics such as accent. Similar problems were also investigated using HuBERT and wav2vec 2.0 models in [261]. In [262] domain effects were studied in greater detail using datasets from six different domains.…”

ESSumm: Extractive Speech Summarization from Untranscribed Meeting

“…Recently, there are several works focusing on utilizing the pre-trained speech representations on different speech-related tasks, including speech recognition [27,28] and speech enhancement [29]. Compared with the traditional handcrafted features, including pause, duration, fundamental frequency (F0), and Mel Frequency Cepstral Coefficients (MFCCs), a variety of pre-trained representations obtain significant improvements in the task of speech recognition [27]. Specifically, the recent Wav2Vec2.0 [28] is a transformer-based speech framework that is trained by predicting speech units for masked parts of the audio.…”