System building cost vs. output quality in data-to-text generation

Abstract

Data-to-text generation systems tend to be knowledge-based and manually built, which limits their reusability and makes them time and cost-intensive to create and maintain. Methods for automating (part of) the system building process exist, but do such methods risk a loss in output quality? In this paper, we investigate the cost/quality trade-off in generation system building. We compare four new data-to-text systems which were created by predominantly automatic techniques against six existing systems for the same domain which were created by predominantly manual techniques. We evaluate the ten systems using intrinsic automatic metrics and human quality ratings. We find that increasing the degree to which system building is automated does not necessarily result in a reduction in output quality. We find furthermore that standard automatic evaluation metrics underestimate the quality of handcrafted systems and over-estimate the quality of automatically created systems.

From data to text in the Neonatal Intensive Care Unit: Using NLG technology for decision support and information management

“…For example, SumTime [105,112] was shown to generate forecasts which were preferred by human readers over those produced by professional forecasters. This was arguably the first such evaluation, and has been followed up by others in a related vein [10,11].…”

Assessing the Trade-Off between System Building Cost and Output Quality in Data-to-Text Generation

“…In material not previously reported in Belz & Kow 2009 [5], we also present the results of some follow-up experiments involving two additional, newly built systems, which we carried out in order to look into the impact of alternative input representations on the fully automatically trainable systems (Section 7), and present some further discussion of (i) issues in assessing the extent to which a system has been built manually, and (ii) the implications of the discrepancies we see between automatic and humanassessed evaluations (Section 8).…”

Statistical Natural Language Generation from Tabular Non-textual Data

“…For both human and automatic evaluation, we compared our system with ten existing NLG systems whose outputs on the Prodigy-METEO testset are also available in the Prodigy-METEO corpus. These ten NLG systems are PCFG-Greedy, PSCFG-Semantic, PSCFG-Unstructured, PCFG-Viterbii, PCFG2gram, PCFG-Roulette, PBSMT-Unstructured, SumTime-Hybrid, PBSMT-Structured and PCFGRandom (Belz and Kow, 2009). Figure 4 shows a sample input and outputs of all the above mentioned systems including our system.…”

“…They used the MOSES 1 toolkit (Koehn et al, 2007) for this purpose. Belz and Kow (2009) proposed another SMT based NLG system which made use of the phrase-based SMT (PB-SMT) model (Koehn et al, 2003). The MOSES toolkit offers an efficient implementation of the PB-SMT model.…”