How Do Seq2Seq Models Perform on End-to-End Data-to-Text Generation?

Abstract

With the rapid development of deep learning, Seq2Seq paradigm has become prevalent for end-to-end data-to-text generation, and the BLEU scores have been increasing in recent years. However, it is widely recognized that there is still a gap between the quality of the texts generated by models and the texts written by human. In order to better understand the ability of Seq2Seq models, evaluate their performance and analyze the results, we choose to use Multidimensional Quality Metric(MQM) to evaluate several representative Seq2Seq models on end-to-end data-to-text generation. We annotate the outputs of five models on four datasets with eight error types and find that 1) copy mechanism is helpful for the improvement in Omission and Inaccuracy Extrinsic errors but it increases other types of errors such as Addition; 2) pre-training techniques are highly effective, and pre-training strategy and model size are very significant; 3) the structure of the dataset also influences the model's performance greatly; 4) some specific types of errors are generally challenging for seq2seq models.

Using knowledge graphs and behaviour trees for feedback-aware presentation agents

“…When generating text for fictional data, the most common error was instead information missing from the generated text. Yin and Wan (2022) showed that the large pretrained T5 and BART models performed practically no addition or duplication errors on any of the KG datasets they were evaluated on, but that the rates of hallucinations (intrinsic and extrinsic inaccuracy) rose with the amount of pretraining -our results on ChatGPT do not follow this trend, as we see both types of errors on our factual dataset.…”

“…For each of the seven triples in the prompt, annotators also had to check one of three exclusive options; the text states this fact (henceforth present), the text does not say anything about this (absent) or the text states something else that actively goes against this fact (hallucinated). Absent corresponds to omission in Yin and Wan (2022), with hallucinated corresponding to inaccuracy intrinsic, inaccuracy extrinsic and positive-negative aspect (Yin and Wan, 2022).…”

“…Although LLMs appear to be able to do a relatively good job at generating text expressing arbitrary KG data, the relatively high rate of inserted hallucinated information (around 10-15% in Section 4.3.3) means system designers must be careful before deploying any system using a LLM KG-to-text as a synthesis engine or microplanner. The rate of addition previously seen in (Yin and Wan, 2022) has one outlier of an approximate rate of 13%, but most of the high-performance models also seen in the WebNLG 2020 challenge (Castro Ferreira et al, 2020) have a much lower rate of inserting information (Yin and Wan, 2022). This suggests ChatGPT is unusually likely to make these types of mistakes.…”

“…The process of converting data represented as knowledge graphs into text is sometimes referred to as graph to text (Schmitt et al, 2020;, or KG to text (Schmitt et al, 2021;Wang et al, 2019). The term Data-to-Text typically refers to a more general group of tasks of which KG-to-text is part (Nan et al, 2021;Yin and Wan, 2022;Ji et al, 2023). Competitions like the WebNLG 2020 challenge contained tracks for both KG-to-text and text-to-KG (Castro Ferreira et al, 2020), but this paper only considers the KG-to-text task.…”

TaTa: A Multilingual Table-to-Text Dataset for African Languages

“…17 This finding is intuitive as these metrics were not designed to evaluate the correctness of reasoning. Nevertheless, they are used to assess outputs in recent data-to-text approaches (e.g.,Mehta et al, 2022;Yin and Wan, 2022; Anders et al, 2022), although most point out the limitations of such automatic assessments.…”