BinaryBERT: Pushing the Limit of BERT Quantization

Bai, Haoli; Zhang, Wei; Hou, Lu; Shang, Lifeng; Jin, Jin; Jiang, Xin; Li, Qun; Lyu, Michael R.; King, Irwin

doi:10.18653/v1/2021.acl-long.334

Cited by 71 publications

(94 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this case, it potentially results in more carbon emission due to more training time on GPU servers, especially when training with larger models and datasets. Future works may consider combining gradient quantization [1] or utilize lower-bits quantization [3,49] to achieve better training efficiency. To understand the effect of different λ in Mesa, we conduct experiments with DeiT-Ti on CIFAR-100 and report the results in Table 9.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Mesa: A Memory-saving Training Framework for Transformers

Pan¹,

Chen²,

He³

et al. 2021

Preprint

View full text Add to dashboard Cite

There has been an explosion of interest in designing high-performance Transformers. While Transformers have delivered significant performance improvements, training such networks is extremely memory intensive owing to storing all intermediate activations that are needed for gradient computation during backpropagation, especially for long sequences. To this end, we present Mesa, a memorysaving resource-efficient training framework for Transformers. Specifically, Mesa uses exact activations during forward pass while storing a low-precision version of activations to reduce memory consumption during training. The low-precision activations are then dequantized during backpropagation to compute gradients. Besides, to address the heterogeneous activation distributions in the multi-head self-attention layers, we propose a head-wise activation quantization strategy, which quantizes activations based on the statistics of each head to minimize the approximation error. To further boost training efficiency, we learn quantization parameters by running estimates. More importantly, by re-investing the saved memory in employing a larger batch size or scaling up model size, we may further improve the performance under constrained computational resources. Extensive experiments on ImageNet, CIFAR-100 and ADE20K demonstrate that Mesa can reduce half of the memory footprints during training while achieving comparable or even better performance. Code is available at https://github.com/zhuang-group/Mesa.

show abstract

Section: Discussionmentioning

confidence: 99%

“…In Transformers, the majority of the literature belongs to the first category. For example, 8-bit [39,48] or even lower-bits [3] quantization has been proposed to speed up the inference. In contrast, this paper focuses on training Transformers from scratch.…”

Section: Introductionmentioning

confidence: 99%

Mesa: A Memory-saving Training Framework for Transformers

Pan¹,

Chen²,

He³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, Bhandare et al (2019) and Prato et al (2020) showed that 8-bit quantization can successfully reduce the size of a Transformer-based model and accelerate inference without compromising translation quality. Recently, quantization has been applied on Transformerbased language models (Zafrir et al, 2019;Bai et al, 2020;. Zafrir et al (2019) first applied 8-bit quantization on BERT.…”

Section: Quantizationmentioning

confidence: 99%

A Survey on Green Deep Learning

Xu¹,

Zhou²,

Fu³

et al. 2021

Preprint

View full text Add to dashboard Cite

In recent years, larger and deeper models are springing up and continuously pushing state-of-the-art (SOTA) results across various fields like natural language processing (NLP) and computer vision (CV). However, despite promising results, it needs to be noted that the computations required by SOTA models have been increased at an exponential rate. Massive computations not only have a surprisingly large carbon footprint but also have negative effects on research inclusiveness and deployment on real-world applications. Green deep learning is an increasingly hot research field that appeals to researchers to pay attention to energy usage and carbon emission during model training and inference. The target is to yield novel results with lightweight and efficient technologies. Many technologies can be used to achieve this goal, like model compression and knowledge distillation. This paper focuses on presenting a systematic review of the development of Green deep learning technologies. We classify these approaches into four categories: (1) compact networks, (2) energy-efficient training strategies, (3) energy-efficient inference approaches, and (4) efficient data usage. For each category, we discuss the progress that has been achieved and the unresolved challenges.

show abstract

“…Pruning and quantization are widely applied techniques for compressing deep neural networks prior to deployment, as compressed models require less memory, energy consumption and have lower inference latency (Esteva et al, 2017;Lane and Warden, 2018;Sun et al, 2020). To-date, evaluating the merits and tradeoffs incurred by compression have overwhelmingly centered on settings where the data is relatively abundant Li et al, 2020a;Chen et al, 2021;Bai et al, 2020;ab Tessera et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation

Ahia¹,

Kreutzer²,

Hooker³

2021

Preprint

View full text Add to dashboard Cite

A "bigger is better" explosion in the number of parameters in deep neural networks has made it increasingly challenging to make stateof-the-art networks accessible in computerestricted environments. Compression techniques have taken on renewed importance as a way to bridge the gap. However, evaluation of the trade-offs incurred by popular compression techniques has been centered on high-resource datasets. In this work, we instead consider the impact of compression in a data-limited regime. We introduce the term low-resource double bind to refer to the co-occurrence of data limitations and compute resource constraints. This is a common setting for NLP for low-resource languages, yet the trade-offs in performance are poorly studied. Our work offers surprising insights into the relationship between capacity and generalization in data-limited regimes for the task of machine translation. Our experiments on magnitude pruning for translations from English into Yoruba, Hausa, Igbo and German show that in low-resource regimes, sparsity preserves performance on frequent sentences but has a disparate impact on infrequent ones. However, it improves robustness to out-of-distribution shifts, especially for datasets that are very distinct from the training distribution. Our findings suggest that sparsity can play a beneficial role at curbing memorization of low frequency attributes, and therefore offers a promising solution to the low-resource double bind.

show abstract

BinaryBERT: Pushing the Limit of BERT Quantization

Cited by 71 publications

References 29 publications

Mesa: A Memory-saving Training Framework for Transformers

Mesa: A Memory-saving Training Framework for Transformers

A Survey on Green Deep Learning

The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation

Contact Info

Product

Resources

About