DART: Open-Domain Structured Data Record to Text Generation

Nan, Linyong; Radev, Dragomir; Zhang, Rui; Rau, Amrit; Sivaprasad, Abhinand; Hsieh, Chia‐Chun; Tang, Xiangru; Vyas, Aadit; Verma, Neha; Krishna, Pranav; Liu, Yangxiaokang; Irwanto, Nadia; Pan, Janet L.; Rahman, Faiaz; Zaidi, Ahmad; Mutuma, Mutethia; Tarabar, Yasin; Gupta, Ankit; Yu, Tao; Tan, Yi Chern; Lin, Xi Victoria; Xiong, Caiming; Socher, Richard; Rajani, Nazneen Fatema

doi:10.18653/v1/2021.naacl-main.37

Cited by 49 publications

(69 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Next, we study private fine-tuning for text generation problems using the GPT-2 series of models on the End-2-End (E2E) NLG challenge (Novikova et al, 2017) and DART (Nan et al, 2021), two primary benchmarks used in recent works on non-private fine-tuning (Hu et al, 2021;. We use GPT-2-Small (117M parameters), GPT-2-Medium (345M parameters), GPT-2-Large (774M parameters), and GPT-2-XL (1.5B parameters).…”

Section: Fine-tuning For Language Understanding Tasksmentioning

confidence: 99%

See 1 more Smart Citation

Differentially Private Fine-tuning of Language Models

Yu¹,

Naik²,

Bačkurs³

et al. 2021

Preprint

View full text Add to dashboard Cite

We give simpler, sparser, and faster algorithms for differentially private fine-tuning of large-scale pre-trained language models, which achieve the state-of-the-art privacy versus utility tradeoffs on many standard NLP tasks. We propose a meta-framework for this problem, inspired by the recent success of highly parameter-efficient methods for fine-tuning. Our experiments show that differentially private adaptations of these approaches outperform previous private algorithms in three important dimensions: utility, privacy, and the computational and memory cost of private training. On many commonly studied datasets, the utility of private models approaches that of non-private models. For example, on the MNLI dataset we achieve an accuracy of 87.8% using RoBERTa-Large and 83.5% using RoBERTa-Base with a privacy budget of ε = 6.7. In comparison, absent privacy constraints, RoBERTa-Large achieves an accuracy of 90.2%. Our findings are similar for natural language generation tasks. Privately fine-tuning with DART, GPT-2-Small, GPT-2-Medium, GPT-2-Large, and GPT-2-XL achieve BLEU scores of 38.5, 42.0, 43.1, and 43.8 respectively (privacy budget of ε = 6.8, δ = 1e-5) whereas the non-private baseline is 48.1. All our experiments suggest that larger models are better suited for private fine-tuning: while they are well known to achieve superior accuracy non-privately, we find that they also better maintain their accuracy when privacy is introduced.

show abstract

Section: Fine-tuning For Language Understanding Tasksmentioning

confidence: 99%

“…DART: DART was introduced as an open-domain data-to-text dataset by Nan et al (2021). The dataset consists of 62K training samples, 6.9K validation samples, and 12K test samples.…”

Section: Fine-tuning For Language Understanding Tasksmentioning

confidence: 99%

Differentially Private Fine-tuning of Language Models

Yu¹,

Naik²,

Bačkurs³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Fine-tuning for Graph-to-text Generation. While previous approaches (Song et al, 2018;Ribeiro et al, 2019;Cai and Lam, 2020;Schmitt et al, 2021;Zhang et al, 2020b) have shown that explicitly encoding the graph structure is beneficial, fine-tuning PLMs on linearized structured data has established a new level of performance in data-to-text generation (Nan et al, 2021;Kale, 2020;. Our work can be seen as integrating the advantage of both graph structure encoding and PLMs, using a novel adapter module.…”

Section: Related Workmentioning

confidence: 98%

Structural Adapters in Pretrained Language Models for AMR-to-Text Generation

Ribeiro¹,

Zhang²,

Gurevych³

2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Pretrained language models (PLM) have recently advanced graph-to-text generation, where the input graph is linearized into a sequence and fed into the PLM to obtain its representation. However, efficiently encoding the graph structure in PLMs is challenging because such models were pretrained on natural language, and modeling structured data may lead to catastrophic forgetting of distributional knowledge. In this paper, we propose STRUCTADAPT, an adapter method to encode graph structure into PLMs. Contrary to prior work, STRUCTADAPT effectively models interactions among the nodes based on the graph connectivity, only training graph structure-aware adapter parameters. In this way, we incorporate task-specific knowledge while maintaining the topological structure of the graph. We empirically show the benefits of explicitly encoding graph structure into PLMs using STRUCTADAPT, outperforming the state of the art on two AMR-to-text datasets, training only 5.1% of the PLM parameters. 1

show abstract

“…Data-to-Text As shown in Figure 5, we fine-tune T5 (Raffel et al, 2019) on DART (Nan et al, 2021) to obtain a Data-to-Text model as the second module of the pipeline to perform surface realization of table cells (denotations in our case). We first convert the denotation prediction into the triple-set format with the following scheme: for each table cell in the highlighted region, we generate the following triple: [[TABLECONTEXT], column header, cell value], where column header is the cell's corresponding column name.…”

Section: Weakly Supervised Table Semantic Parsingmentioning

confidence: 99%

“…We use a checkpoint of TAPAS-base that is fine-tuned on WikiTableQuestions (Pasupat and Liang, 2015) to perform table semantic parsing implicitly in order to produce a set of denotations, which is then converted to a triple-set as described in 3.1. We then employ a T5-large model (Raffel et al, 2019) that goes through two fine-tuning stages: in the first stage it is fine-tuned on the downstream Data-to-Text task with DART (Nan et al, 2021); in the second stage it is further fine-tuned on ToTTo instances to adapt to the triple-set formulation we proposed. We denote this setting as Pipeline -zeroshot in Table 4.…”

Section: Experiments Setupmentioning

confidence: 99%

FeTaQA: Free-form Table Question Answering

Nan¹,

Hsieh²,

Mao³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Existing table question answering datasets contain abundant factual questions that primarily evaluate the query and schema comprehension capability of a system, but they fail to include questions that require complex reasoning and integration of information due to the constraint of the associated short-form answers. To address these issues and to demonstrate the full challenge of table question answering, we introduce Fe-TaQA, a new dataset with 10K Wikipediabased {table, question, free-form answer, supporting table cells} pairs. FeTaQA yields a more challenging table question answering setting because it requires generating free-form text answers after retrieval, inference, and integration of multiple discontinuous facts from a structured knowledge source. Unlike datasets of generative QA over text in which answers are prevalent with copies of short text spans from the source, answers in our dataset are human-generated explanations involving entities and their high-level relations. We provide two benchmark methods for the proposed task: a pipeline method based on semantic parsingbased QA systems and an end-to-end method based on large pretrained text generation models, and show that FeTaQA poses a challenge for both methods.

show abstract

DART: Open-Domain Structured Data Record to Text Generation

Cited by 49 publications

References 36 publications

Differentially Private Fine-tuning of Language Models

Differentially Private Fine-tuning of Language Models

Structural Adapters in Pretrained Language Models for AMR-to-Text Generation

FeTaQA: Free-form Table Question Answering

Contact Info

Product

Resources

About