DARE: Data Augmented Relation Extraction with GPT-2

Papanikolaou, Yannis; Pierleoni, Andrea

doi:10.48550/arxiv.2004.13845

Cited by 18 publications

(18 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent works show that large pretrained LMs are capable of generating data of reasonable quality (Anaby-Tavor et al, 2020;Papanikolaou and Pierleoni, 2020;Yang et al, 2020;Mohapatra et al, 2021;Kumar et al, 2020;Schick and Schütze, 2021a;Meng et al, 2022), sometimes leading to better transfer learning than human generated datasets (Liu et al, 2022).…”

Section: Related Workmentioning

confidence: 99%

InPars: Data Augmentation for Information Retrieval using Large Language Models

Bonifacio¹,

Abonizio²,

Fadaee³

et al. 2022

Preprint

View full text Add to dashboard Cite

The information retrieval community has recently witnessed a revolution due to large pretrained transformer models. Another key ingredient for this revolution was the MS MARCO dataset, whose scale and diversity has enabled zero-shot transfer learning to various tasks. However, not all IR tasks and domains can benefit from one single dataset equally. Extensive research in various NLP tasks has shown that using domain-specific training data, as opposed to a general-purpose one, improves the performance of neural models (Yu et al., 2021;Sharami et al., 2022). In this work, we harness the fewshot capabilities of large pretrained language models as synthetic data generators for IR tasks. We show that models finetuned solely on our unsupervised dataset outperform strong baselines such as BM25 as well as recently proposed selfsupervised dense retrieval methods. Furthermore, retrievers finetuned on both supervised and our synthetic data achieve better zero-shot transfer than models finetuned only on supervised data. Code, models, and data are available at https: //github.com/zetaalphavector/inpars

show abstract

Section: Related Workmentioning

confidence: 99%

InPars: Data Augmentation for Information Retrieval using Large Language Models

Bonifacio¹,

Abonizio²,

Fadaee³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Synthetic data generation with GPT-2 is also explored for relation extraction in Papanikolaou and Pierleoni (2020). This paper fine-tunes GPT-2 over labeled examples of the same relation type, where each sentence in the training data is marked with the two entity mentions in the corresponding relation.…”

Section: Information Extraction (Ie)mentioning

confidence: 99%

Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey

Min¹,

Ross²,

Sulem³

et al. 2021

Preprint

View full text Add to dashboard Cite

Large, pre-trained transformer-based language models such as BERT have drastically changed the Natural Language Processing (NLP) field. We present a survey of recent work that uses these large language models to solve NLP tasks via pre-training then fine-tuning, prompting, or text generation approaches. We also present approaches that use pre-trained language models to generate data for training augmentation or other purposes. We conclude with discussions on limitations and suggested directions for future research.

show abstract

“…It has demonstrated remarkable zero-shot multitask adaptability by simply feeding the input of each task into the LM and continuing to generate words. People have also also shown that GPT-2 is able to improve classification tasks via in-domain text augmentation (Papanikolaou and Pierleoni, 2020;Sun et al, 2020). We use a similar technique by first finetuning GPT-2 in the few-shot annotations (Wolf et al, 2019), and then applying it to produce synthetic text through an iterative conditional generation process: With initial seeds being samples of T L plus new samples from information augmentation, the LM iteratively conditions on the previous output sentence to generate in-domain text 2 .…”

Section: Lm Augmentationmentioning

confidence: 99%