BAM! Born-Again Multi-Task Networks for Natural Language Understanding

Clark, Kevin; Luong, Minh-Thang; Khandelwal, Urvashi; Manning, Christopher D.; Le, Quoc V.

doi:10.18653/v1/p19-1595

Cited by 188 publications

(176 citation statements)

References 34 publications

Supporting

Mentioning

176

Contrasting

Order By: Relevance

“…As a result, knowledge distillation has become another key feature in this new learning paradigm. An effective distillation step can often substantially compress a large model for efficient deployment (Clark et al, 2019;Tang et al, 2019;Liu et al, 2019a).…”

Section: Introductionmentioning

confidence: 99%

Multi-Task Deep Neural Networks for Natural Language Understanding

Liu¹,

He²,

Chen³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

926

713

View full text Add to dashboard Cite

We present MT-DNN 1 , an open-source natural language understanding (NLU) toolkit that makes it easy for researchers and developers to train customized deep learning models. Built upon PyTorch and Transformers, MT-DNN is designed to facilitate rapid customization for a broad spectrum of NLU tasks, using a variety of objectives (classification, regression, structured prediction) and text encoders (e.g., RNNs, BERT, RoBERTa, UniLM). A unique feature of MT-DNN is its built-in support for robust and transferable learning using the adversarial multi-task learning paradigm. To enable efficient production deployment, MT-DNN supports multitask knowledge distillation, which can substantially compress a deep neural model without significant performance drop. We demonstrate the effectiveness of MT-DNN on a wide range of NLU applications across general and biomedical domains. The software and pretrained models will be publicly available at https://github.com/namisan/mt-dnn.

show abstract

Section: Introductionmentioning

confidence: 99%

Multi-Task Deep Neural Networks for Natural Language Understanding

Liu¹,

He²,

Chen³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

926

713

View full text Add to dashboard Cite

show abstract

“…These efforts have tried multiple teacher distillation methods in the field of computer vision, but little research has been done on the NLP deep pre-training based model. Concurrently with our work, several works also combine the multi-task learning with knowledge distillation [2,18,19]. However, they applied the knowledge distillation and multi-task learning to enhance the original model performance, instead of targeting model compression.…”

Section: Multi-task Learningmentioning

confidence: 86%

“…Therefore, it has captured a rough language model from large corpus. 2 The distillation pre-trained model of stage 1 will be released soon.…”

Section: Tmkd Architecturementioning

confidence: 99%

“…We take the word pieces as the input just like BERT. X = {x (1) , x (2) , ..., x (|X |) } is to denote all the instances, and each instance has a ⟨Q, P⟩ pair. Let Q = {w 1 , w 2 , w 3 , ..., w m } be a question with m word pieces, P = {w 1 , w 2 , w 3 , ..., w n } be a passage with n word pieces, and w i is the bag-of-word representation of i-th word piece.…”

Section: Encodermentioning

confidence: 99%

See 1 more Smart Citation

Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System

Yang

Shou

Gong

et al. 2020

Proceedings of the 13th International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

Deep pre-training and fine-tuning models (such as BERT and Ope-nAI GPT) have demonstrated excellent results in question answering areas. However, due to the sheer amount of model parameters, the inference speed of these models is very slow. How to apply these complex models to real business scenarios becomes a challenging but practical problem. Previous model compression methods usually suffer from information loss during the model compression procedure, leading to inferior models compared with the original one. To tackle this challenge, we propose a Two-stage Multi-teacher Knowledge Distillation (TMKD for short) method for web Question Answering system. We first develop a general Q&A distillation task for student model pre-training, and further fine-tune this pretrained student model with multi-teacher knowledge distillation on downstream tasks (like Web Q&A task, MNLI, SNLI, RTE tasks from GLUE), which effectively reduces the overfitting bias in individual teacher models, and transfers more general knowledge to the student model. The experiment results show that our method can significantly outperform the baseline methods and even achieve comparable results with the original teacher models, along with substantial speedup of model inference.

show abstract

“…Recent work on multi-task learning has focused on designing effective neural architectures (Hashimoto et al, 2017;Søgaard and Goldberg, 2016;Sanh et al, 2018;Ruder et al, 2017). Combining these two lines of work, Clark et al, 2019) explored fine-tuning the contextualized models with multiple natural language understanding tasks. In this work, we depart from previous works by specifically studying the effects of multi-task fine-tuning for the stance prediction task with pre-trained models.…”

Section: Related Workmentioning

confidence: 99%

Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER)

2019

View full text Add to dashboard Cite

We present the results of the second Fact Extraction and VERification (FEVER2.0) Shared Task. The task challenged participants to both build systems to verify factoid claims using evidence retrieved from Wikipedia and to generate adversarial attacks against other participant's systems. The shared task had three phases: building, breaking and fixing. There were 8 systems in the builder's round, three of which were new qualifying submissions for this shared task, and 5 adversaries generated instances designed to induce classification errors and one builder submitted a fixed system which had higher FEVER score and resilience than their first submission. All but one newly submitted systems attained FEVER scores higher than the best performing system from the first shared task and under adversarial evaluation, all systems exhibited losses in FEVER score. There was a great variety in adversarial attack types as well as the techniques used to generate the attacks, In this paper, we present the results of the shared task and a summary of the systems, highlighting commonalities and innovations among participating systems.

show abstract

BAM! Born-Again Multi-Task Networks for Natural Language Understanding

Cited by 188 publications

References 34 publications

Multi-Task Deep Neural Networks for Natural Language Understanding

Multi-Task Deep Neural Networks for Natural Language Understanding

Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering System

Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER)

Contact Info

Product

Resources

About