Patient Knowledge Distillation for BERT Model Compression

Abstract

Pre-trained language models such as BERT have proven to be highly effective for natural language processing (NLP) tasks. However, the high demand for computing resources in training such models hinders their application in practice. In order to alleviate this resource hunger in large-scale model training, we propose a Patient Knowledge Distillation approach to compress an original large model (teacher) into an equally-effective lightweight shallow network (student). Different from previous knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model patiently learns from multiple intermediate layers of the teacher model for incremental knowledge extraction, following two strategies: (i) PKD-Last: learning from the last k layers; and (ii) PKD-Skip: learning from every k layers. These two patient distillation schemes enable the exploitation of rich information in the teacher's hidden layers, and encourage the student model to patiently learn from and imitate the teacher through a multilayer distillation process. Empirically, this translates into improved results on multiple NLP tasks with significant gain in training efficiency, without sacrificing model accuracy. 1

TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing

“…Before the birth of BERT, KD had been applied to several specific tasks like machine translation (Kim and Rush, 2016;Tan et al, 2019) in NLP. While the recent studies of distilling large pre-trained models focus on finding general distillation methods that work on various tasks and are receiving more and more attention (Sanh et al, 2019;Jiao et al, 2019;Sun et al, 2019a;Tang et al, 2019;Clark et al, 2019;.…”

ScopeIt: Scoping Task Relevant Sentences in Documents

“…Furthermore, using BERT for inference poses latency challenges in a production system. A promising direction of future work that we plan on investigating is leveraging distilled versions of BERT (Sun et al, 2019;Wang et al, 2020) for the task. Table 6 shows the results of the NN analysis.…”

Sound Natural: Content Rephrasing in Dialog Systems

“…As such, we look into Knowledge Distillation (KD) (Hinton et al, 2015) to transfer the language modeling capability of BART while keeping its copying behavior. Transferring the language model of massive pre-trained models into smaller models has been of high interest recently (Sanh et al, 2019;Turc et al, 2020;Sun et al, 2019). Knowledge transfer to simple models has also been discussed in lesser extent (Tang et al, 2019;Mukherjee and Awadallah, 2019).…”