TernaryBERT: Distillation-aware Ultra-low Bit BERT

Abstract

Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks. However, these models are both computation and memory expensive, hindering their deployment to resource-constrained devices. In this work, we propose TernaryBERT, which ternarizes the weights in a fine-tuned BERT model. Specifically, we use both approximation-based and loss-aware ternarization methods and empirically investigate the ternarization granularity of different parts of BERT. Moreover, to reduce the accuracy degradation caused by the lower capacity of low bits, we leverage the knowledge distillation technique (Jiao et al., 2019) in the training process. Experiments on the GLUE benchmark and SQuAD show that our proposed TernaryBERT outperforms the other BERT quantization methods, and even achieves comparable performance as the fullprecision model while being 14.9x smaller.

BinaryBERT: Pushing the Limit of BERT Quantization

“…While BinaryBERT focuses on weight binarization, we also explore activation quantization in our implementation, which is beneficial for reducing the computation burden on specialized hardwares (Hubara et al, 2016;Zhou et al, 2016;Zhang et al, 2020). Aside from 8-bit uniform quantization (Zhang et al, 2020; in past efforts, we further pioneer to study 4-bit activation quantization. We find that uniform quantization can hardly deal with outliers in the activation.…”

“…Quantization Details. Following (Zhang et al, 2020), for each weight matrix in the Transformer layers, we use layer-wise ternarization (i.e., one scaling parameter for all elements in the weight matrix). For word embedding, we use row-wise ternarization (i.e., one scaling parameter for each row in the embedding).…”

“…Knowledge distillation is shown to benefit BERT quantization (Zhang et al, 2020). Following (Jiao et al, 2020;Zhang et al, 2020), we first perform intermediate-layer distillation from the fullprecision teacher network's embedding E, layerwise MHA output M l and FFN output F l to the quantized student counterpartÊ,M l ,F l (l = 1, 2, ...L). We aim to minimize their mean sqaured errors, i.e.,…”

“…We report the mean results with standard deviations from 10 seeds on MRPC and 3 seeds on MNLI-m, respectively. Fan et al, 2020;Zhang et al, 2020). Among all these model compression approaches, quantization is a popular solution as it does not require designing a smaller model architecture.…”

TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference

“…Researchers have made various attempts to accelerate the inference of PLMs, such as quantization (Shen et al, 2020;Zhang et al, 2020a), attention head pruning (Michel et al, 2019;, dimension reduction (Sun et al, 2020;Chen et al, 2020), and layer reduction (Sanh et al, 2019;Sun et al, 2019b;Jiao et al, 2019). In current studies, one of the mainstream methods is to dynamically select the layer number of Transformer layers to make a on-demand lighter model (Fan et al, 2020;.…”