ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval

Abstract

With the development of large language models (LLMs), zero-shot learning has attracted much attention for various NLP tasks. Different from prior works that generate training data with billion-scale natural language generation (NLG) models, we propose a retrievalenhanced framework to create training data from a general-domain unlabeled corpus. To realize this, we first conduct contrastive pretraining to learn an unsupervised dense retriever for extracting most relevant documents using classdescriptive verbalizers. We then further propose two simple strategies, namely Verbalizer Augmentation with Demonstrations and Selfconsistency Guided Filtering to improve the topic coverage of the dataset while removing noisy examples. Experiments on nine datasets demonstrate that REGEN achieves 4.3% gain over strongest baselines and saves around 70% of the time when compared with baselines using large NLG models. Besides, REGEN can be naturally integrated with recently proposed large language models to boost performance 1 .

LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing

“…Some participants highlighted that reliance on LLM suggestions might inadvertently cause them to overlook certain videos they would have considered had they worked independently. Moreover, biases present in LLMs during their training phase [22,70,82] have the potential to subtly influence users' creative endeavors. Therefore, it is important to carefully consider the potential for bias introduced by LLMs in the creative process and take steps to mitigate it.…”

A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4

“…The research community explored GLLMs for data generation-based data augmentation in various NLP tasks like dialogue generation [410], training smaller LLMs [411], [416], common sense reasoning [412], hate speech detection [413], undesired content detection [414], question answering [415], [425], intent classification [143], relation extraction [155], [422], instruction tuning [417], [418], paraphrase detection [420], tweet intimacy prediction [421], named entity recognition [422], machine translation [424] etc. GLLM-based data generation for data augmentation is explored in multiple domains like general [143], [155], [412], [416]- [418], [420], [424]- [426], social media [409], [413], [414], [421], [423], news [423], scientific literature [155], [420], healthcare [410], [415], [422], dialogue [419], programming [411] etc. Table 19 presents a summary of research works exploring GLLMs for data generationbased data augmentation.…”

“…Table 19 presents a summary of research works exploring GLLMs for data generationbased data augmentation. Some of the research works explored GLLMs for data generation-based data augmentation in various text classification tasks [143], [409], [413], [414], [421], [423]. For example, Hartvigsen et al [413] used GPT-3 with demonstration-based prompting to create a large-scale synthetic dataset for the detection of implicit hate speech.…”

“…Most of the existing research works use simple prompts for data generation, limiting the diversity of the generated synthetic data. To address this, Yu et al [423] proposed a novel approach that leverages attributed prompts for data generation to increase the diversity in the generated data. Based on the evaluation on four topic classification datasets, the authors observed that (i) the proposed approach enhances the model performance and (ii) reduces the querying cost of ChatGPT by a large margin.…”