Shrinking Bigfoot: Reducing wav2vec 2.0 footprint

Peng, Zilun; Budhkar, Akshay; Tuil, Ilana; Levy, Jason; Sobhani, Parinaz; Cohen, Raphael; Nassour, Jumana

doi:10.18653/v1/2021.sustainlp-1.14

Cited by 13 publications

(2 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reducing architectural complexity of large-scale pretrained models has become an indispensable research endeavor [15,8,16,17,18,19,20,21]. DistilHuBERT [8] is proposed to distill hidden representations from HuBERT BASE.…”

Section: Speech Pre-trained Model Compressionmentioning

confidence: 99%

LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT

Wang¹,

Bai²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Self-supervised speech representation learning has shown promising results in various speech processing tasks. However, the pre-trained models, e.g., HuBERT, are storage-intensive Transformers, limiting their scope of applications under lowresource settings. To this end, we propose LightHuBERT, a once-for-all Transformer compression framework, to find the desired architectures automatically by pruning structured parameters. More precisely, we create a Transformer-based supernet that is nested with thousands of weight-sharing subnets and design a two-stage distillation strategy to leverage the contextualized latent representations from HuBERT. Experiments on automatic speech recognition (ASR) and the SU-PERB benchmark show the proposed LightHuBERT enables over 10 9 architectures concerning the embedding dimension, attention dimension, head number, feed-forward network ratio, and network depth. LightHuBERT outperforms the original HuBERT on ASR and five SUPERB tasks with the Hu-BERT size, achieves comparable performance to the teacher model in most tasks with a reduction of 29% parameters, and obtains a 3.5× compression ratio in three SUPERB tasks, e.g., automatic speaker verification, keyword spotting, and intent classification, with a slight accuracy loss. The code and pre-trained models are available at https://github.com/ mechanicalsea/lighthubert.

show abstract

Section: Speech Pre-trained Model Compressionmentioning

confidence: 99%

LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT

Wang¹,

Bai²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…One common approach is knowledge distillation [11], which trains a small student model with a pre-specified architecture to match the soft targets generated by a large pre-trained model. Distillation has shown to be effective in natural language processing (NLP) [12,13] and speech processing [14,15,16,17], but it usually performs general distillation using large amounts of unlabeled data before task-specific distillation or fine-tuning. This can make the training procedure computationally expensive.…”

Section: Introductionmentioning

confidence: 99%

Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding

Peng¹,

Kim²,

Wu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Self-supervised speech representation learning (SSL) has shown to be effective in various downstream tasks, but SSL models are usually large and slow. Model compression techniques such as pruning aim to reduce the model size and computation without degradation in accuracy. Prior studies focus on the pruning of Transformers; however, speech models not only utilize a stack of Transformer blocks, but also combine a frontend network based on multiple convolutional layers for low-level feature representation learning. This frontend has a small size but a heavy computational cost. In this work, we propose three task-specific structured pruning methods to deal with such heterogeneous networks. Experiments on LibriSpeech and SLURP show that the proposed method is more accurate than the original wav2vec2-base with 10% to 30% less computation, and is able to reduce the computation by 40% to 50% without any degradation.

show abstract