PCEE-BERT: Accelerating BERT Inference via Patient and Confident Early Exiting

Abstract

BERT and other pre-trained language models (PLMs) are ubiquitous in modern NLP. Even though PLMs are the state-of-the-art (SOTA) models for almost every NLP task (Qiu et al., 2020), the significant latency during inference prohibits wider industrial usage. In this work, we propose Patient and Confident Early Exiting BERT (PCEE-BERT), an off-the-shelf sample-dependent early exiting method that can work with different PLMs and can also work along with popular model compression methods. With a multi-exit BERT as the backbone model, PCEE-BERT will make the early exiting decision if enough numbers (patience parameter) of consecutive intermediate layers are confident about their predictions. The entropy value measures the confidence level of an intermediate layer's prediction. Experiments on the GLUE benchmark demonstrate that our method outperforms previous SOTA early exiting methods. Ablation studies show that: (a) our method performs consistently well on other PLMs, such as ALBERT and TinyBERT; (b) PCEE-BERT can achieve different speed-up ratios by adjusting the patience parameter and the confidence threshold. The code for PCEE-BERT can be found at https://github. com/michael-wzhu/PCEE-BERT.

ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inference

“…In PABEE (Zhou et al 2020), the instance exits when k consecutive internal classifiers make the same prediction. PCEE-BERT (Zhang et al 2022) combined both ensemble-based and confidence-based exiting criteria. The instance exits if the confidence scores are greater than a predefined threshold for several consecutive exits.…”

“…The Thirty-Eighth AAAI Conference on Artificial Intelligence the sum of cross-entropy losses. SkipBERT (Wang et al 2022), PABEE (Zhou et al 2020), Past-Future (Liao et al 2021), PCEE-BERT (Zhang et al 2022), and LeeBERT (Zhu 2021)) used weighted sum of cross entropy losses. Dee-BERT (Xin et al 2020), Right-Tool (Schwartz et al 2020), BERxiT (Xin et al 2021), and CAT (Schuster et al 2021) used sum of cross entropy losses.…”

“…Statistics of these datasets are listed in Table 2. We compared our method with various baselines including Dee-BERT (Xin et al 2020), PABEE (Zhou et al 2020), BERxiT (Xin et al 2021), Right-Tool (Schwartz et al 2020), PCEE-BERT (Zhang et al 2022), HashEE (Sun et al 2022), and TR-BERT (Ye et al 2021). For layer-wise exiting methods, we use saved layers to evaluate model acceleration.…”

“…2023), and knowledge distillation (Sanh et al 2019;Sun et al 2019;Jiao et al 2020). Dynamic approaches (Xu and McAuley 2023) include token skipping (Goyal et al 2020;Kim and Cho 2021;Ye et al 2021;Kim et al 2022;Guan et al 2022;Sun et al 2022), and early exiting (Zhou et al 2020;Xin et al 2021;Zhang et al 2022).…”

Leap-of-Thought: Accelerating Transformers via Dynamic Token Routing

“…PABEE has proposed a patiencebased exit strategy that halts the forward-pass at an intermediate layer only when the pre-defined number of subsequent layers yield the same predictions. Similarly, DeeBERT and FastBERT have employed the predictive entropy to replace the patience, and PCEE-BERT (Zhang et al, 2022) has combined both patience and confidence for the exit criteria.…”