Investigating Pretrained Language Models for Graph-to-Text Generation

Abstract

Graph-to-text generation aims to generate fluent texts from graph-based data. In this paper, we investigate two recent pretrained language models (PLMs) and analyze the impact of different task-adaptive pretraining strategies for PLMs in graph-to-text generation. We present a study across three graph domains: meaning representations, Wikipedia knowledge graphs (KGs) and scientific KGs. We show that approaches based on PLMs BART and T5 achieve new state-of-the-art results and that task-adaptive pretraining strategies improve their performance even further. We report new state-of-the-art BLEU scores of 49.72 on AMR-LDC2017T10, 59.70 on WebNLG, and 25.66 on AGENDA datasets -a relative improvement of 31.8%, 4.5%, and 42.4%, respectively, with our models generating significantly more fluent texts than human references. In an extensive analysis, we identify possible reasons for the PLMs' success on graph-totext tasks. Our findings suggest that the PLMs benefit from similar facts seen during pretraining or fine-tuning, such that they perform well even when the input graph is reduced to a simple bag of node and edge labels. 1

WDV: A Broad Data Verbalisation Dataset Built from Wikidata

“…Working with single-claim data, we do not need to maintain the graph's structure. Large pre-trained language models have achieved state of the art results when fine-tuned and evaluated on WebNLG [33,16,15], mainly the T5 [30]; they can disregard most structure and can be applied to one or many claims at a time. Hence, we utilise the T5 (base version) as our verbalisation model, following training and evaluation methods from these works.…”

“…The SEEN partition is used for both training and validation/testing, while the UNSEEN partition is kept for testing only. We follow the training setup from Ribeiro et al [33] by specifying a new prefix "translate from Graph to Text" and adding three new tokens ( H , R , and T ) that precede the claim's subject, predicate, and object, respectively.…”

“…Firstly, by evaluating aggregated crowdsourced metrics, covered in Section 3.5. Secondly, by calculating the BLEU score [28] for both the SEEN and UNSEEN partitions, to make sure our model matches current state of the art [33]. BLEU scores were calculated with Moses [21].…”

“…Comparisons between the model employed by us (T5-base), other language models, and other mentioned approaches on verbalising the WebNLG dataset have been carried by other authors [33,3] and is out of the scope of our paper.…”

“…Data verbalisation, a facet of Natural Language Generation (NLG), is a task that has great importance in the current field of natural language processing [33,16,15,36,45,11], as there is great benefit in the transformation of our abundant structured and semi-structured data into human-readable formats. It is important in its own right, as well as as a step towards larger tasks such as open-domain question-answering [24] and automated fact checking [41,42].…”