A Snapshot of NLG Evaluation Practices 2005 - 2014

Gkatzia, Dimitra; Mahamood, Saad

doi:10.18653/v1/w15-4708

Cited by 47 publications

(56 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This paper shows that state-of-the-art automatic evaluation metrics for NLG systems do not sufficiently reflect human ratings, which stresses the need for human evaluations. This result is opposed to the current trend of relying on automatic evaluation identified in (Gkatzia and Mahamood, 2015).…”

Section: Discussioncontrasting

confidence: 65%

Why We Need New Evaluation Metrics for NLG

Novikova¹,

Dušek²,

Curry³

et al. 2017

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

343

276

View full text Add to dashboard Cite

The majority of NLG evaluation relies on automatic metrics, such as BLEU. In this paper, we motivate the need for novel, system-and data-independent automatic evaluation methods: We investigate a wide range of metrics, including state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judgements of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data-and system-specific. Nevertheless, our results also suggest that automatic metrics perform reliably at system-level and can support system development by finding cases where a system performs poorly.

show abstract

Section: Discussioncontrasting

confidence: 65%

Why We Need New Evaluation Metrics for NLG

Novikova¹,

Dušek²,

Curry³

et al. 2017

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

343

276

View full text Add to dashboard Cite

show abstract

“…Only three papers (3%) in the sample of INLG and ACL papers presented an extrinsic evaluation. This is a notable decrease from Gkatzia and Mahamood (2015), who found that nearly 25% of studies contained an extrinsic evaluation. Of course, extrinsic evaluation is the most time-and cost-intensive out of all possible evaluations (Gatt and Krahmer, 2018), which might explain the rarity, but does not explain the decline in (relative) frequency.…”

Section: Intrinsic and Extrinsic Evaluationmentioning

confidence: 87%

“…Previous studies have also provided overviews of evaluation methods. Gkatzia and Mahamood (2015) focused on NLG papers from 2005-2014; Amidei et al (2018a) provided a 2013-2018 overview of evaluation in question generation; and Gatt and Krahmer (2018) provided a more general survey of the state-of-the-art in NLG. However, the aim of these papers was to give a structured overview of existing methods, rather than discuss shortcomings and best practices.…”

Section: Introductionmentioning

confidence: 99%

Best practices for the human evaluation of automatically generated text

Lee

Gatt

Miltenburg

et al. 2019

Proceedings of the 12th International Conference on Natural Language Generation

125

View full text Add to dashboard Cite

Currently, there is little agreement as to how Natural Language Generation (NLG) systems should be evaluated, with a particularly high degree of variation in the way that human evaluation is carried out. This paper provides an overview of how human evaluation is currently conducted, and presents a set of best practices, grounded in the literature. With this paper, we hope to contribute to the quality and consistency of human evaluations in NLG.

show abstract

“…This includes a novel application of textual measures 18 and a novel usage of standard word-overlap metrics to assess similarity among individual systems. Automatic metrics are popular in NLG (Gkatzia and Mahamood, 2015) because they are cheaper and faster to run than human evaluation. However, sole use of automatic metrics is only sensible if they are known to be sufficiently correlated with human preferences.…”

Section: Evaluation Setupmentioning

confidence: 99%

Evaluating the state-of-the-art of End-to-End Natural Language Generation: The E2E NLG challenge

Dušek

Novikova

Rieser

2020

Computer Speech & Language

174

183

View full text Add to dashboard Cite

This paper provides a comprehensive analysis of the first shared task on End-to-End Natural Language Generation (NLG) and identifies avenues for future research based on the results. This shared task aimed to assess whether recent end-to-end NLG systems can generate more complex output by learning from datasets containing higher lexical richness, syntactic complexity and diverse discourse phenomena. Introducing novel automatic and human metrics, we compare 62 systems submitted by 17 institutions, covering a wide range of approaches, including machine learning architectures -with the majority implementing sequence-to-sequence models (seq2seq) -as well as systems based on grammatical rules and templates. Seq2seq-based systems have demonstrated a great potential for NLG in the challenge. We find that seq2seq systems generally score high in terms of word-overlap metrics and human evaluations of naturalness -with the winning Slug system (Juraska et al., 2018) being seq2seq-based.However, vanilla seq2seq models often fail to correctly express a given meaning representation if they lack a strong semantic control mechanism applied during decoding.Moreover, seq2seq models can be outperformed by hand-engineered systems in terms of overall quality, as well as complexity, length and diversity of outputs. This research has influenced, inspired and motivated a number of recent studies outwith the original competition, which we also summarise as part of this paper.

show abstract

A Snapshot of NLG Evaluation Practices 2005 - 2014

Cited by 47 publications

References 2 publications

Why We Need New Evaluation Metrics for NLG

Why We Need New Evaluation Metrics for NLG

Best practices for the human evaluation of automatically generated text

Evaluating the state-of-the-art of End-to-End Natural Language Generation: The E2E NLG challenge

Contact Info

Product

Resources

About