Evaluating factual accuracy in complex data-to-text

Thomson, Craig; Reiter, Ehud; Sundararajan, Barkavi

doi:10.1016/j.csl.2023.101482

Cited by 19 publications

(24 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thomson and Reiter (2021) noted that even within a single criterion, two annotators could provide sets of errors that only partially overlap, yet can both be considered valid representations of the same complex underlying problem. In addition to calculating agreement, annotators could check each other's annotations and indicate whether they consider them one valid way of describing the underlying problems Thomson et al (2023).…”

Section: Error Taxonomies and Standardizationmentioning

confidence: 99%

Barriers and enabling factors for error analysis in NLG research

Miltenburg

Clinciu²,

Dušek³

et al. 2023

NEJLT

Self Cite

View full text Add to dashboard Cite

Earlier research has shown that few studies in Natural Language Generation (NLG) evaluate their system outputs using an error analysis, despite known limitations of automatic evaluation metrics and human ratings. This position paper takes the stance that error analyses should be encouraged, and discusses several ways to do so. This paper is not just based on our shared experience as authors, but we also distributed a survey as a means of public consultation. We provide an overview of existing barriers to carry out error analyses, and proposes changes to improve error reporting in the NLG literature.

show abstract

Section: Error Taxonomies and Standardizationmentioning

confidence: 99%

Barriers and enabling factors for error analysis in NLG research

Miltenburg

Clinciu²,

Dušek³

et al. 2023

NEJLT

Self Cite

View full text Add to dashboard Cite

show abstract

“…They should be provided with the annotation guidelines ( §4.5), and then be asked to annotate texts where the errors are known (but not visible). The solutions would ideally be created by experts, although in some cases, solutions created by researchers may be sufficient (Thomson and Reiter, 2020). It should be decided in advance what the threshold is to accept annotators for the reaming work, and, if they fail, whether to provide additional training or find other candidates.…”

Section: Number Of Annotatorsmentioning

confidence: 99%

“…For example, Costa et al (2015) describe how different taxonomies of errors in Machine Translation build on each other. In NLG, if you are working on data-to-text, then you could take Thomson and Reiter's (2020) taxonomy as a starting point. Alternatively, Dou et al (2021) present a crowd-sourced error annotation called SCARECROW.…”

Section: Bottom-up Approaches First Identify Differentmentioning

confidence: 99%

“…For instance, we may award partial credit (1/n error categories), mark both types of errors as applying in this situation (overgeneralising, to be on the safe side), or count all ambiguous cases to separately report on them in the overall frequency table. Another solution, used by Thomson and Reiter (2020) is to provide the annotators with a fixed preference order (NAME, NUMBER, WORD, CONTEXT), so that similar cases are resolved in a similar fashion.…”

Section: Bottom-up Approaches First Identify Differentmentioning

confidence: 99%

“…A notable exception is the paper by Thomson and Reiter (2020), who carry out an error analysis of existing output data from three different systems. This paper was not considered amenable, because…”

Section: A Annotationmentioning

confidence: 99%

See 2 more Smart Citations

Underreporting of errors in NLG output, and what to do about it

van Miltenburg,

Clinciu,

Dušek

et al. 2021

Proceedings of the 14th International Conference on Natural Language Generation

View full text Add to dashboard Cite

We observe a severe under-reporting of the different kinds of errors that Natural Language Generation systems make. This is a problem, because mistakes are an important indicator of where systems should still be improved. If authors only report overall performance metrics, the research community is left in the dark about the specific weaknesses that are exhibited by 'state-of-the-art' research. Next to quantifying the extent of error under-reporting, this position paper provides recommendations for error identification, analysis and reporting.

show abstract

Introduction to NLG

Reiter

2024

Natural Language Generation

View full text Add to dashboard Cite

Evaluating factual accuracy in complex data-to-text

Cited by 19 publications

References 31 publications

Barriers and enabling factors for error analysis in NLG research

Barriers and enabling factors for error analysis in NLG research

Underreporting of errors in NLG output, and what to do about it

Introduction to NLG

Contact Info

Product

Resources

About