A Study of Automatic Metrics for the Evaluation of Natural Language Explanations

Clinciu, Miruna-Adriana; Eshghi, Arash; Hastie, Helen

doi:10.18653/v1/2021.eacl-main.202

Cited by 26 publications

(21 citation statements)

References 58 publications

(63 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Automatic (and human) evaluation processes are well-known problems for the field of Natural Language Generation (Howcroft et al, 2020;Clinciu et al, 2021) and the burgeoning subfield of ST is not immune. ST, in particular, has suffered from a lack of standardization of automatic metrics, a lack of agreement between human judgments and automatics metrics, as well as a blindspot to developing metrics for languages other than English.…”

Section: Discussionmentioning

confidence: 99%

Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer

Briakou¹,

Agrawal²,

Tetreault³

et al. 2021

Preprint

View full text Add to dashboard Cite

While the field of style transfer (ST) has been growing rapidly, it has been hampered by a lack of standardized practices for automatic evaluation. In this paper, we evaluate leading ST automatic metrics on the oft-researched task of formality style transfer. Unlike previous evaluations, which focus solely on English, we expand our focus to Brazilian-Portuguese, French, and Italian, making this work the first multilingual evaluation of metrics in ST. We outline best practices for automatic evaluation in (formality) style transfer and identify several models that correlate well with human judgments and are robust across languages. We hope that this work will help accelerate development in ST, where human evaluation is often challenging to collect.

show abstract

Section: Discussionmentioning

confidence: 99%

Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer

Briakou¹,

Agrawal²,

Tetreault³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…11 We only ask Questions 2 and 3 if the answer to Question 1 is "yes" because they regard the new facts, information, or reasoning. We found that most prior work tends to lump added-value, relevance, and adequacy judgements into one "informativeness" judgement (Clinciu et al, 2021), which we felt was too course to allow for meaningful error analysis.…”

Section: B2 Absolute Interface Detailsmentioning

confidence: 93%

“…Figures 5 and 6 show the absolute evaluation interface. Our interface is inspired by prior work from psychology and the social sciences (Leake, 1991;Gopnik, 1998;Lombrozo, 2007;Zemla et al, 2017;Chiyah Garcia et al, 2018;Clinciu et al, 2021;Sulik et al, 2021). We iterated over 3-4 versions of the questions and UI design until we had optimized agreement rates as much as possible.…”

Section: B2 Absolute Interface Detailsmentioning

confidence: 99%

See 1 more Smart Citation

Reframing Human-AI Collaboration for Generating Free-Text Explanations

Wiegreffe¹,

Hessel²,

Swayamdipta³

et al. 2021

Preprint

View full text Add to dashboard Cite

Large language models are increasingly capable of generating fluent-appearing text with relatively little task-specific supervision. But can these models accurately explain classification decisions? We consider the task of generating free-text explanations using a small number of human-written examples (i.e., in a few-shot manner). We find that (1) authoring higher-quality examples for prompting results in higher quality generations; and(2) surprisingly, in a head-to-head comparison, crowdworkers often prefer explanations generated by GPT-3 to crowdsourced humanwritten explanations contained within existing datasets. Crowdworker ratings also show, however, that while models produce factual, grammatical, and sufficient explanations, they have room to improve, e.g., along axes such as providing novel information and supporting the label. We create a pipeline that combines GPT-3 with a supervised filter that incorporates humans-in-the-loop via binary acceptability judgments. Despite significant subjectivity intrinsic to judging acceptability, our approach is able to consistently filter GPT-3 generated explanations deemed acceptable by humans.

show abstract

“…In contrast, we additionally evaluate commonsense inclusion as well as grammatical correctness of explanations. As Clinciu et al (2021) find automatic BLEURT scores to have distinctly stronger correlations to human ratings of generated textual explanations than BLEU, we investigate whether BLEURT is a viable replacement for a user study.…”

Section: Evaluation and Human Ratingsmentioning

confidence: 99%

Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratings

Schuff¹,

Yang²,

Adel³

et al. 2021

Preprint

View full text Add to dashboard Cite

Natural language inference (NLI) requires models to learn and apply commonsense knowledge. These reasoning abilities are particularly important for explainable NLI systems that generate a natural language explanation in addition to their label prediction. The integration of external knowledge has been shown to improve NLI systems, here we investigate whether it can also improve their explanation capabilities. For this, we investigate different sources of external knowledge and evaluate the performance of our models on in-domain data as well as on special transfer datasets that are designed to assess finegrained reasoning capabilities. We find that different sources of knowledge have a different effect on reasoning abilities, for example, implicit knowledge stored in language models can hinder reasoning on numbers and negations. Finally, we conduct the largest and most fine-grained explainable NLI crowdsourcing study to date. It reveals that even large differences in automatic performance scores do neither reflect in human ratings of label, explanation, commonsense nor grammar correctness.

show abstract

A Study of Automatic Metrics for the Evaluation of Natural Language Explanations

Cited by 26 publications

References 58 publications

Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer

Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer

Reframing Human-AI Collaboration for Generating Free-Text Explanations

Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratings

Contact Info

Product

Resources

About