Neural Pipeline for Zero-Shot Data-to-Text Generation

Kasner, Zdeněk; Dušek, Ondřej

doi:10.18653/v1/2022.acl-long.271

Cited by 18 publications

(22 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 14 shows the recall/precision on different error types, for the bestperforming metric overall (Kasner et al, 2021). We can see that it was unable 805 to detect CONTEXT C , NOT CHECKABLE X , and OTHER O , and only had around 50% precision and recall for WORD W errors.…”

Section: Resultsmentioning

confidence: 99%

“…• Kasner et al (2021) developed an automatic metric which used a rulebased system and a semantic similarity filter to produce known-to-beaccurate sentences which are similar to the sentence being evaluated for 780 accuracy. A model was then trained to detect accuracy errors, using as input both the sentence being assessed and the known-to-be-accurate sentences.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Evaluating factual accuracy in complex data-to-text

Thomson

Reiter

Sundararajan

2023

Computer Speech & Language

View full text Add to dashboard Cite

Section: Resultsmentioning

confidence: 99%

mentioning

confidence: 99%

Evaluating factual accuracy in complex data-to-text

Thomson

Reiter

Sundararajan

2023

Computer Speech & Language

View full text Add to dashboard Cite

“…Third, we added one paper ) from the related work cited by Al , and four relevant papers we were already aware of Kasner and Dusek, 2022;Popović, 2020), the last of these as a (rare) example of work using the top-level content/meaning error type (Adequacy, Accuracy, see Section) in annotation.…”

Section: Paper Selection and Filteringmentioning

confidence: 99%

“…The latter distinguishes errors on a number of different dimensions, of which however just two are used in the reported work: intrinsic (misrepresented words from the source text) vs. extrinsic (added words not in the source text) errors, involving a noun phrase vs. a predicate. Kasner and Dusek (2022) present a zero-shot alternative for data-to-text generation using ordering, aggregation, and paragraph compression. A manual error analysis is performed using five error types: Hallucination, Incorrect Fact Merging, Omissions, Redundancy, Grammar Error, and Disfluency.…”

Section: Summaries Of Papersmentioning

confidence: 99%

Towards a Consensus Taxonomy for Annotating Errors in Automatically Generated Text

Huidrom,

Belz

2023

Proceedings of the Conference Recent Advances in Natural Language Processing - Large Language Models for Natural Language Proce

View full text Add to dashboard Cite

Error analysis aims to provide insights into system errors at different levels of granularity. NLP as a field has a long-standing tradition of analysing and reporting errors which is generally considered good practice. There are existing error taxonomies tailored for different types of NLP task. In this paper, we report our work reviewing existing research on meaning/content error types in generated text, attempt to identify emerging consensus among existing meaning/content error taxonomies, and propose a standardised error taxonomy on this basis. We find that there is virtually complete agreement at the highest taxonomic level where errors of meaning/content divide into (1) Content Omission, (2) Content Addition, and (3) Content Substitution. Consensus in the lower levels is less pronounced, but a compact standardised consensus taxonomy can nevertheless be derived that works across generation tasks and application domains.

show abstract

“…Previous studies on hallucination detection have primarily concentrated on identifying hallucinations produced by small models (fewer than 1b parameters) that are tailored for specific tasks. For instance, Kasner et al (2021) combined a rulebased system and a pretrained language model to identify hallucinations in table-to-text generation. Guerreiro et al (2022) adopted the average log-probability across all the tokens in the output sequence as the model uncertainty metric for detecting hallucinations in machine translation.…”

Section: Hallucination Detectionmentioning

confidence: 99%

Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus

Zhang,

Qiu,

Guo

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Large Language Models (LLMs) have gained significant popularity for their impressive performance across diverse fields. However, LLMs are prone to hallucinate untruthful or nonsensical outputs that fail to meet user expectations in many real-world applications. Existing works for detecting hallucinations in LLMs either rely on external knowledge for reference retrieval or require sampling multiple responses from the LLM for consistency verification, making these methods costly and inefficient. In this paper, we propose a novel referencefree, uncertainty-based method for detecting hallucinations in LLMs. Our approach imitates human focus in factuality checking from three aspects: 1) focus on the most informative and important keywords in the given text; 2) focus on the unreliable tokens in historical context which may lead to a cascade of hallucinations; and 3) focus on the token properties such as token type and token frequency. Experimental results on relevant datasets demonstrate the effectiveness of our proposed method, which achieves state-of-the-art performance across all the evaluation metrics and eliminates the need for additional information. 1

show abstract

Neural Pipeline for Zero-Shot Data-to-Text Generation

Cited by 18 publications

References 0 publications

Evaluating factual accuracy in complex data-to-text

Evaluating factual accuracy in complex data-to-text

Towards a Consensus Taxonomy for Annotating Errors in Automatically Generated Text

Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus

Contact Info

Product

Resources

About