MC-BERT: Efficient Language Pre-Training via a Meta Controller

Xu, Zhenhui; Gong, Linyuan; Ke, Guolin; He, Di; Zheng, Shuxin; Wang, Liwei; Bian, Jiang; Liu, Tie-Yan

doi:10.48550/arxiv.2006.05744

Cited by 6 publications

(13 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are inevitable variations in the pretraining corpora used in different work. Our base corpus is obtained from the authors of MC-BERT (Xu et al, 2020) and TUPE (Ke et al, 2020). Our base++ corpus is the most similar with those used in UniLM (Dong et al, 2019;Bao et al, 2020).…”

Section: Methodsmentioning

confidence: 99%

“…Implementation Details: Our implementation is built upon the open-source release of MC-BERT (Xu et al, 2020) and its ELECTRA reproduction based on fairseq (Ott et al, 2019). Standard hyperparameters in pretraining and finetuning are used.…”

Section: Methodsmentioning

confidence: 99%

“…Although it underperforms ELECTRA on GLUE, ELEC-TRIC still pretrains a language model which can be used in tasks like scoring the feasibility of a generated text sequence. MC-BERT uses a multiple-choice task which selects original tokens from plausible alternatives and performs on par with ELECTRA (Xu et al, 2020). COCO-LM also leverages an auxiliary network to generate pretraining inputs but uses two new pretraining tasks, one of which is a language modeling task.…”

Section: Related Workmentioning

confidence: 99%

“…The randomly chosen masks do not always provide the best pretraining signals: Many masked tokens are trivial common words that may not push the Transformer to capture meaningful language semantics (Guu et al, 2020), while some might be too hard or have many false negatives (Xu et al, 2020). Pretraining on those masks is not guaranteed to elevate the language model's generalization ability.…”

Section: Preliminarymentioning

confidence: 99%

“…The baseline results listed in Table 1 are obtained from their original papers except the following: BERT from (Bao et al, 2020), RoBERTa GLUE from and SQuAD from (Bao et al, 2020), ELECTRA GLUE from (Xu et al, 2020), XLNet base++ from (Bao et al, 2020), RoBERTa base++ SQuAD from (Bao et al, 2020). When multiple papers report different scores for the same method, we use the highest of them in our comparisons.…”

Section: The Origins Of Reported Baseline Scoresmentioning

confidence: 99%

See 4 more Smart Citations

COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

Yu¹,

Xiong²,

Bajaj³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present COCO-LM, a new self-supervised learning framework that pretrains Language Models by COrrecting challenging errors and COntrasting text sequences. COCO-LM employs an auxiliary language model to mask-and-predict tokens in original text sequences. It creates more challenging pretraining inputs, where noises are sampled based on their likelihood in the auxiliary language model. COCO-LM then pretrains with two tasks: The first task, corrective language modeling, learns to correct the auxiliary model's corruptions by recovering the original tokens. The second task, sequence contrastive learning, ensures that the language model generates sequence representations that are invariant to noises and transformations. In our experiments on the GLUE and SQuAD benchmarks, COCO-LM outperforms recent pretraining approaches in various pretraining settings and few-shot evaluations, with higher pretraining efficiency. Our analyses reveal that COCO-LM's advantages come from its challenging training signals, more contextualized token representations, and regularized sequence representations.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Preliminarymentioning

confidence: 99%

Section: The Origins Of Reported Baseline Scoresmentioning

confidence: 99%

See 3 more Smart Citations

COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

Yu¹,

Xiong²,

Bajaj³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Training ELECTRA Augmented with Multi-word Selection

Shen¹,

Liu²,

Liu³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

Pre-trained text encoders such as BERT and its variants have recently achieved state-of-the-art performances on many NLP tasks. While being effective, these pre-training methods typically demand massive computation resources. To accelerate pre-training, ELECTRA trains a discriminator that predicts whether each input token is replaced by a generator. However, this new task, as a binary classification, is less semantically informative. In this study, we present a new text encoder pre-training method that improves ELECTRA based on multi-task learning. Specifically, we train the discriminator to simultaneously detect replaced tokens and select original tokens from candidate sets. We further develop two techniques to effectively combine all pre-training tasks: (1) using attention-based networks for task-specific heads, and (2) sharing bottom layers of the generator and the discriminator. Extensive experiments on GLUE and SQuAD datasets demonstrate both the effectiveness and the efficiency of our proposed method. * This work is done while interning at Google Research.

show abstract