BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

Xu, Canwen; Zhou, Wangchunshu; Ge, Tao; Wei, Furu; Zhou, Ming

doi:10.48550/arxiv.2002.02925

Cited by 21 publications

(25 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…CompressingBERT [50] Quantization Q-BERT [150], Q8BERT [204] Parameter Sharing ALBERT [91] Distillation DistilBERT [146], TinyBERT [74], MiniLM [188] Module Replacing BERT-of-Theseus [196] Figure 3: Taxonomy of PTMs with Representative Examples Sentence Order Prediction (SOP) To better model intersentence coherence, ALBERT [91] replaces the NSP loss with a sentence order prediction (SOP) loss. As conjectured in Lan et al [91], NSP conflates topic prediction and coherence prediction in a single task.…”

Section: Model Pruningmentioning

confidence: 99%

“…Module replacing is an interesting and simple way to reduce the model size, which replaces the large modules of original PTMs with more compact substitutes. Xu et al [196] proposed Theseus Compression motivated by a famous thought experiment called "Ship of Theseus", which progressively substitutes modules from the source model with modules of fewer parameters. Different from KD, Theseus Compression only requires one task-specific loss function.…”

Section: Module Replacingmentioning

confidence: 99%

“…Comparison of Compressed PTMs The desing of this table is borrowed from[196,144].‡ The averaged score on 8 tasks (without WNLI) of GLUE benchmark (see Section7.1). Here MNLI-m and MNLI-mm are regarded as two different tasks.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Pre-trained Models for Natural Language Processing: A Survey

Qiu,

Sun,

et al. 2020

Preprint

View full text Add to dashboard Cite

Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy with four perspectives. Next, we describe how to adapt the knowledge of PTMs to the downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.

show abstract

Section: Model Pruningmentioning

confidence: 99%

Section: Module Replacingmentioning

confidence: 99%

See 1 more Smart Citation

Pre-trained Models for Natural Language Processing: A Survey

Qiu,

Sun,

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…In addition, remarkable advances have been made in knowledge distillation for language model compression (i.e. , BERT [13]), and these works show that mimicking the distribution of self-attention and intermediate representations of transformer blocks increases performances [52,27,58,69] for downstream tasks. In particular, in the transformer-based language model distillation, Dis-tillBERT [52] proposes to train the small BERT by mimicking the Teacher's output probability of masked language prediction and the embedding features.…”

Section: Related Workmentioning

confidence: 99%

Compressing Visual-linguistic Model via Knowledge Distillation

Fang

Wang

et al. 2021

Preprint

View full text Add to dashboard Cite

Despite exciting progress in pre-training for visuallinguistic (VL) representations, very few aspire to a small VL model. In this paper, we study knowledge distillation (KD) to effectively compress a transformer based large VL model into a small VL model. The major challenge arises from the inconsistent regional visual tokens extracted from different detectors of Teacher and Student, resulting in the misalignment of hidden representations and attention distributions. To address the problem, we retrain and adapt the Teacher by using the same region proposals from Student's detector while the features are from Teacher's own object detector. With aligned network inputs, the adapted Teacher is capable of transferring the knowledge through the intermediate representations. Specifically, we use the mean square error loss to mimic the attention distribution inside the transformer block, and present a token-wise noise contrastive loss to align the hidden state by contrasting with negative representations stored in a sample queue. To this end, we show that our proposed distillation significantly improves the performance of small VL models on image captioning and visual question answering tasks. It reaches 120.8 in CIDEr score on COCO captioning, an improvement of 5.1 over its non-distilled counterpart; and an accuracy of 69.8 on VQA 2.0, a 0.8 gain from the baseline. Our extensive experiments and ablations confirm the effectiveness of VL distillation in both pre-training and fine-tuning stages.

show abstract

“…MobileBERT takes approximately 0.6 seconds to classify a text sequence on a Google Pixel 3 smartphone. And, on the GLUE benchmark, which consists of 9 natural language understanding (NLU) datasets [16], MobileBERT achieves higher accuracy than other efficient networks such as DistilBERT [17], PKD [18], and several others [19,20,21,22]. To achieve this, MobileBERT introduced two concepts into their NLP self-attention network that are already in widespread use in CV neural networks:…”

Section: What Has CV Research Already Taught Nlp Research About Effic...mentioning

confidence: 99%

SqueezeBERT: What can computer vision teach NLP about efficient neural networks?

Iandola¹,

Shaw²,

Krishna³

et al. 2020

Preprint

View full text Add to dashboard Cite

Humans read and write hundreds of billions of messages every day. Further, due to the availability of large datasets, large computing systems, and better neural network models, natural language processing (NLP) technology has made significant strides in understanding, proofreading, and organizing these messages. Thus, there is a significant opportunity to deploy NLP in myriad applications to help web users, social networks, and businesses. In particular, we consider smartphones and other mobile devices as crucial platforms for deploying NLP models at scale. However, today's highly-accurate NLP neural network models such as BERT and RoBERTa are extremely computationally expensive, with BERT-base taking 1.7 seconds to classify a text snippet on a Pixel 3 smartphone. In this work, we observe that methods such as grouped convolutions have yielded significant speedups for computer vision networks, but many of these techniques have not been adopted by NLP neural network designers. We demonstrate how to replace several operations in self-attention layers with grouped convolutions, and we use this technique in a novel network architecture called SqueezeBERT, which runs 4.3x faster than BERT-base on the Pixel 3 while achieving competitive accuracy on the GLUE test set. The SqueezeBERT code will be released.

show abstract

BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

Cited by 21 publications

References 24 publications

Pre-trained Models for Natural Language Processing: A Survey

Pre-trained Models for Natural Language Processing: A Survey

Compressing Visual-linguistic Model via Knowledge Distillation

SqueezeBERT: What can computer vision teach NLP about efficient neural networks?

Contact Info

Product

Resources

About