The Lottery Ticket Hypothesis for Pre-trained BERT Networks

Abstract

In natural language processing (NLP), enormous pre-trained models like BERT have become the standard starting point for training on a range of downstream tasks, and similar trends are emerging in other areas of deep learning. In parallel, work on the lottery ticket hypothesis has shown that models for NLP and computer vision contain smaller matching subnetworks capable of training in isolation to full accuracy and transferring to other tasks. In this work, we combine these observations to assess whether such trainable, transferrable subnetworks exist in pre-trained BERT models. For a range of downstream tasks, we indeed find matching subnetworks at 40% to 90% sparsity. We find these subnetworks at (pre-trained) initialization, a deviation from prior NLP research where they emerge only after some amount of training. Subnetworks found on the masked language modeling task (the same task used to pre-train the model) transfer universally; those found on other tasks transfer in a limited fashion if at all. As large-scale pre-training becomes an increasingly central paradigm in deep learning, our results demonstrate that the main lottery ticket observations remain relevant in this context. Codes available at https://github.com/TAMU-VITA/BERT-Tickets.

On the Compression of Natural Language Models

“…On the other hand, in structured pruning, the best subnetworks of BERT's heads do not quite reach the full model performance. (Chen et al, 2020a) shows for a range of downstream tasks, matching subnetworks at 40% to 90% sparsity exist and they are found at pretrained phase (initialization). This is dissimilar to the prior NLP research where subnetworks emerge only after some amount of training.…”

“…In view of that, follow-up works reveal that sparsity patterns might emerge at the initialization , the early stage of training (You et al, 2019) and (Chen et al, 2020b), or in dynamic forms throughout training (Evci et al, 2020) by updating model parameters and architecture typologies simultaneously. Some of the recent findings are that the lottery ticket hypothesis holds for BERT models, i.e., largest weights of the original network do form subnetworks that can be retrained alone to reach the performance close to that of the full model (Prasanna et al, 2020;Chen et al, 2020a).…”

“…1.6.1 LTH for BERT (Chen et al, 2020a) applies the LTH to identify matching subnetworks in pretrained BERT models to enforce sparsity in models trained for different downstream tasks. They show that assuming pretrained parameters as the initialization for BERT models, one can find sparse subnetworks that can be trained for downstream NLP tasks.…”

“…Experimental evaluation of EarlyBERT shows some degradation in accuracy for fine-tuning. (Prasanna et al, 2020) and (Chen et al, 2020a) explore BERT models from the perspective of the lottery ticket hypothesis (Frankle and Carbin, 2018), looking specifically at the "winning" subnetworks in pre-trained BERT. They find that such subnetworks do exist, and that transferability between subnetworks for different tasks varies.…”

VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer

“…To address this issue, many efforts have been devoted to compressing the cumbersome transformer architectures into a lightweight counterpart, including knowledge distillation (Jiao et al, 2019;Sanh et al, 2019;Sun et al, 2019;, pruning (Michel et al, 2019;Chen et al, 2020;, and quantization (Zafrir et al, 2019;Bai et al, 2020;Shen et al, 2020). Among all these compression techniques, quantization is a popular solution as it still preserves the original network architecture.…”