Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understanding

Li, Shiyang; Yavuz, Semih; Chen, Wenhu; Yan, Xifeng

doi:10.18653/v1/2021.findings-emnlp.86

Cited by 3 publications

(4 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The pseudo-labeled data are either used for self-training (Schick and Schütze, 2021) or consistency training (Xie et al, 2020). All these approaches can also be combined to enable more efficient use of unlabeled data (Li et al, 2021b;Chen et al, 2021;Zhao and Yao, 2022). In our experiments, we found that simple self-training using the same data for DAPT can further improve the performance of AdaSent.…”

Section: Semi-supervised Text Classificationmentioning

confidence: 99%

See 1 more Smart Citation

AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification

Huang,

Wang,

Dutta

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Recent work has found that few-shot sentence classification based on pre-trained Sentence Encoders (SEs) is efficient, robust, and effective. In this work, we investigate strategies for domain-specialization in the context of fewshot sentence classification with SEs. We first establish that unsupervised Domain-Adaptive Pre-Training (DAPT) of a base Pre-trained Language Model (PLM) (i.e., not an SE) substantially improves the accuracy of few-shot sentence classification by up to 8.4 points. However, applying DAPT on SEs, on the one hand, disrupts the effects of their (general-domain) Sentence Embedding Pre-Training (SEPT). On the other hand, applying general-domain SEPT on top of a domain-adapted base PLM (i.e., after DAPT) is effective but inefficient, since the computationally expensive SEPT needs to be executed on top of a DAPT-ed PLM of each domain. As a solution, we propose AdaSent, which decouples SEPT from DAPT by training a SEPT adapter on the base PLM. The adapter can be inserted into DAPT-ed PLMs from any domain. We demonstrate AdaSent's effectiveness in extensive experiments on 17 different few-shot sentence classification datasets. AdaSent matches or surpasses the performance of full SEPT on DAPT-ed PLM, while substantially reducing the training costs. The code for AdaSent is available 1 .

show abstract

Section: Semi-supervised Text Classificationmentioning

confidence: 99%

“…Besides DAPT, another major way to utilize the unlabeled data is self-training, which has been shown to be complementary to DAPT (Li et al, 2021b).…”

Section: Combining Dapt and Self-trainingmentioning

confidence: 99%

AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification

Huang,

Wang,

Dutta

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…Then, individual sentence descriptions are generated separately for each group, and combined to form the final description for the input. In contrast to previous studies that primarily rely on self-training to improve CG in DTG (He et al, 2020;Heidari et al, 2021;Li et al, 2021;Mehta et al, 2022), as well as using data augmentation to improve CG in various tasks such as semantic parsing (Andreas, 2020;Qiu et al, 2022;Fang et al, 2023), our method solely relies on small training sets and does not require any additional human-annotated, automatically labeled, or unlabeled data. In the CG-centric testing scenario, we observe significant improvements across all metrics in the benchmark compared to the vanilla T5 model (Raffel et al, 2020;Kale and Rastogi, 2020b).…”

Section: Introductionmentioning

confidence: 98%

Compositional Generalization for Data-to-Text Generation

Xu,

Titov,

Lapata

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Data-to-text generation involves transforming structured data, often represented as predicateargument tuples, into coherent textual descriptions. Despite recent advances, systems still struggle when confronted with unseen combinations of predicates, producing unfaithful descriptions (e.g., hallucinations or omissions). We refer to this issue as compositional generalisation, and it encouraged us to create a benchmark for assessing the performance of different approaches on this specific problem. Furthermore, we propose a novel model that addresses compositional generalization by clustering predicates into groups. Our model generates text in a sentence-by-sentence manner, relying on one cluster of predicates at a time. This approach significantly outperforms T5 baselines across all evaluation metrics. Notably, it achieved a 31% improvement over T5 in terms of a metric focused on maintaining faithfulness to the input. 1

show abstract

“…Unlike common teacherstudent training for knowledge distillation (Hinton et al, 2015;Gou et al, 2021), here the teacher does not train the student on all the available instances (in our case, all the incoming customer inputs). Also, unlike teacher-student approaches to self-training (Mi et al, 2021;Li et al, 2021), the teacher is already reasonably effective (but expensive). In that sense, our work is closer to ac-tive learning (Settles, 2012;Monarch, 2021), but OCaTS trains the student on labels provided by a teacher LLM, not humans, and there is initially no large pool of unlabeled instances (customer inputs) to select from, as instances arrive online.…”

Section: Introductionmentioning

confidence: 99%

Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models

Stogiannidis,

Vassos,

Malakasiotis

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Prompting Large Language Models (LLMs) performs impressively in zero-and few-shot settings. Hence, small and medium-sized enterprises (SMEs) that cannot afford the cost of creating large task-specific training datasets, but also the cost of pretraining their own LLMs, are increasingly turning to third-party services that allow them to prompt LLMs. However, such services currently require a payment per call, which becomes a significant operating expense (OpEx). Furthermore, customer inputs are often very similar over time, hence SMEs end-up prompting LLMs with very similar instances. We propose a framework that allows reducing the calls to LLMs by caching previous LLM responses and using them to train a local inexpensive model on the SME side. The framework includes criteria for deciding when to trust the local model or call the LLM, and a methodology to tune the criteria and measure the tradeoff between performance and cost. For experimental purposes, we instantiate our framework with two LLMs, GPT-3.5 or GPT-4, and two inexpensive students, a k-NN classifier or a Multi-Layer Perceptron, using two common business tasks, intent recognition and sentiment analysis. Experimental results indicate that significant OpEx savings can be obtained with only slightly lower performance.

show abstract

Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understanding

Cited by 3 publications

References 10 publications

AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification

AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification

Compositional Generalization for Data-to-Text Generation

Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models

Contact Info

Product

Resources

About