Challenges in Data-to-Document Generation

Abstract

Recent neural models have shown significant progress on the problem of generating short descriptive texts conditioned on a small number of database records. In this work, we suggest a slightly more difficult data-to-text generation task, and investigate how effective current approaches are on this task. In particular, we introduce a new, large-scale corpus of data records paired with descriptive documents, propose a series of extractive evaluation methods for analyzing performance, and obtain baseline results using current neural generation methods. Experiments show that these models produce fluent text, but fail to convincingly approximate humangenerated documents. Moreover, even templated baselines exceed the performance of these neural models on some metrics, though copy-and reconstructionbased extensions lead to noticeable improvements.

A Hierarchical Model for Data-to-Text Generation

“…This equation is intractable in practice, we approximate a solution using beam search, as in [18,17,28,29,39].…”

“…The first one (module A in Figure 2) is called low-level encoder and encodes entities on the basis of their records; the second one (module B), called high-level encoder, encodes the data-structure on the basis of its underlying entities. In the low-level encoder, the traditional embedding layer is replaced by a record embedding layer as in [18,28,39]. We present in what follows the record embedding layer and introduce our two hierarchical modules.…”

“…The BLEU score [23] aims at measuring to what extent the generated descriptions are literally closed to the ground truth. The second category designed by [39] is more qualitative.…”

“…Particularly, they compare the gold and generated descriptions and measure to what extent the extracted relations are aligned or differ. To do so, we follow the protocol presented in [39]. First, we apply an information extraction (IE) system trained on labeled relations from the gold descriptions of the RotoWire train dataset.…”

“…The decoder is the one used in [28,29,39] with the same hyper-parameters. For the encoder module, both the low-level and high-level encoders use a two-layers multi-head self-attention with two heads.…”