Self-training Improves Pre-training for Natural Language Understanding

Abstract

Unsupervised pre-training has led to much recent progress in natural language understanding. In this paper, we study self-training as another way to leverage unlabeled data through semi-supervised learning. To obtain additional data for a specific task, we introduce SentAugment, a data augmentation method which computes task-specific query embeddings from labeled data to retrieve sentences from a bank of billions of unlabeled sentences crawled from the web. Unlike previous semisupervised methods, our approach does not require in-domain unlabeled data and is therefore more generally applicable. Experiments show that self-training is complementary to strong RoBERTa baselines on a variety of tasks. Our augmentation approach leads to scalable and effective self-training with improvements of up to 2.6% on standard text classification benchmarks. Finally, we also show strong gains on knowledge-distillation and few-shot learning. * Equal contribution.An alternative semi-supervised technique is pretraining (Dai and Le, 2015;Radford et al., 2018;Howard and Ruder, 2018;Devlin et al., 2018), which has led to large improvements for natural language understanding compared to purely supervised learning. In that case, models are first trained on an auxiliary task, such as language modeling, followed by fine-tuning on the task of interest.A natural question is the following: do pretraining and self-training capture the same information, or are they complementary? Recently, Zoph et al. ( 2020) studied this question in the context of image recognition, showing that selftraining was helpful, even in addition to pretraining. However, their study mostly considers supervised pre-training, in which models were trained on ImageNet classification. Moreover, in cases where large amounts of supervised data were available for the downstream task, pre-training was not helpful, even without self-training. This is in contrast to natural language understanding for which language modeling pre-training is a very strong baseline that leads to large improvements for all the tasks we consider.An important ingredient for self-training, and semi-supervised learning in general, is the unannotated data and the fact that it comes from the same domain as the downstream task. Existing work, such as UDA (Xie et al., 2019), selftraining (He et al., 2019Xie et al., 2020) and back-translation for machine translation (Bojar and Tamchyna, 2011;Sennrich et al., 2015;Edunov et al., 2018), assumes the existence of unannotated data in the same domain as the downstream task. This assumption limits the broad application of such semi-supervised methods, in particular in the case of low-resource downstream tasks. A second important question is thus: how can we obtain large amounts of unannotated data from specific domains?

Streaming Self-Training via Domain-Agnostic Unlabeled Images

“…We present a simple and intuitive approach to semisupervised learning on (potentially) infinite streams of unlabeled data. Our approach integrates insights from different bodies of work including self-training [19,85], pseudolabelling [41,4,35], continual/iterated learning [38,39,74,75,67], and few-shot learning [44,28]. We demonstrate a number of surprising conclusions: (1) Unlabeled domainagnostic internet streams can be used to significantly improve models for specialized tasks and data domains, including surface normal prediction, semantic segmentation, and few-shot fine-grained image classification spanning diverse domains including medical, satellite, and agricultural imagery.…”

“…Self-Training and Semi-Supervised Learning: A large variety of self-training [19,85] and semi-supervised approaches [58,65,78,94,91,93] use unlabeled images in conjunction with labeled images to learn a better representation (Fig. 2-(b)).…”

Improved Text Classification via Contrastive Adversarial Training

“…As a general method to improve Transformer-based model performance on downstream tasks, [36] and [16] propose further language model pretraining in the target domain before the final fine-tuning. [11] proposes to use self-training as another way to leverage unlabeled data, where a teacher model is first trained on labeled data, and is then used to label large amount of in-domain unlabeled data for the student model to learn from. Recent developments in language model pretraining have also advanced state-of-the-art results on a wide range of NLP tasks.…”

Self-Training with Differentiable Teacher

“…The approach has gain popularity in many applications. For example, in conjunction with pre-trained language models (Devlin et al, 2019), self-training has demonstrated superior performance on tasks such as natural language understanding (Du et al, 2020), named entity recognition (Liang et al, 2020), and question answering (Sachan and Xing, 2018).…”

“…Self-training is an effective complement to these pre-trained models. For example, combining self-training with pre-trained language models achieves superior performance in tasks such as text classification (Meng et al, 2020;Mukherjee and Awadallah, 2020;Du et al, 2020), named entity recognition (Liang et al, 2020), reading comprehension (Niu et al, 2020) and dialogue systems (Mi et al, 2021).…”