Interrogating the Explanatory Power of Attention in Neural Machine Translation

Moradi, Pooya; Kambhatla, Nishant; Sarkar, Anoop

doi:10.18653/v1/d19-5624

Cited by 12 publications

(10 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While most of these works provide evidence that attention weights are not always faithful, Moradi et al (2019) confirm similar observations on the unfaithful nature of attention in the context of NMT models. Li et al (2020) is one of the few papers examining attention models in NMT.…”

Section: Related Worksupporting

confidence: 75%

“…It is worth noting that increase in faithfulness of attention-based explanations for function words is much more than that of content words. This can be attributed to the fact the function words are mostly generated using the target-side information in the decoder (Tu et al, 2017;Moradi et al, 2019) and manipulating attention does not have much effect on generating them. However, our proposed faithfulness objective (F f aith ) seems to tighten the dependence of the decoder on the attention component.…”

Section: Impact On Faithfulnessmentioning

confidence: 99%

See 1 more Smart Citation

Measuring and Improving Faithfulness of Attention in Neural Machine Translation

Moradi

Kambhatla²,

Sarkar

2021

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

Self Cite

View full text Add to dashboard Cite

While the attention heatmaps produced by neural machine translation (NMT) models seem insightful, there is little evidence that they reflect a model's true internal reasoning. We provide a measure of faithfulness for NMT based on a variety of stress tests where attention weights which are crucial for prediction are perturbed and the model should alter its predictions if the learned weights are a faithful explanation of the predictions. We show that our proposed faithfulness measure for NMT models can be improved using a novel differentiable objective that rewards faithful behaviour by the model through probability divergence. Our experimental results on multiple language pairs show that our objective function is effective in increasing faithfulness and can lead to a useful analysis of NMT model behaviour and more trustworthy attention heatmaps. Our proposed objective improves faithfulness without reducing the translation quality and has a useful regularization effect on the NMT model and can even improve translation quality in some cases.

show abstract

Section: Related Worksupporting

confidence: 75%

Section: Impact On Faithfulnessmentioning

confidence: 99%

Measuring and Improving Faithfulness of Attention in Neural Machine Translation

Moradi

Kambhatla²,

Sarkar

2021

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

Self Cite

View full text Add to dashboard Cite

show abstract

“…Attention mechanisms learn to assign soft weights to (usually contextualized) token representations, and so one can extract highly weighted tokens as rationales. However, attention weights do not in general provide faithful explanations for predictions (Jain and Wallace, 2019;Serrano and Smith, 2019;Wiegreffe and Pinter, 2019;Zhong et al, 2019;Pruthi et al, 2020;Brunner et al, 2020;Moradi et al, 2019;Vashishth et al, 2019). This likely owes to encoders entangling inputs, complicating the interpretation of attention weights on inputs over contextualized representations of the same.…”

Section: Related Workmentioning

confidence: 99%

ERASER: A Benchmark to Evaluate Rationalized NLP Models

DeYoung

Jain

Rajani³

et al. 2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

319

341

View full text Add to dashboard Cite

State-of-the-art models in NLP are now predominantly based on deep neural networks that are opaque in terms of how they come to make predictions. This limitation has increased interest in designing more interpretable deep models for NLP that reveal the 'reasoning' behind model outputs. But work in this direction has been conducted on different datasets and tasks with correspondingly unique aims and metrics; this makes it difficult to track progress. We propose the Evaluating Rationales And Simple English Reasoning (ERASER ) benchmark to advance research on interpretable models in NLP. This benchmark comprises multiple datasets and tasks for which human annotations of "rationales" (supporting evidence) have been collected. We propose several metrics that aim to capture how well the rationales provided by models align with human rationales, and also how faithful these rationales are (i.e., the degree to which provided rationales influenced the corresponding predictions). Our hope is that releasing this benchmark facilitates progress on designing more interpretable NLP systems. The benchmark, code, and documentation are available at https://www.eraserbenchmark.com/ Commonsense Explanations (CoS-E)Where do you find the most amount of leafs? (a) Compost pile (b) Flowers (c) Forest (d) Field (e) Ground Movie ReviewsIn this movie, … Plots to take over the world. The acting is great! The soundtrack is run-of-the-mill, but the action more than makes up for it (a) Positive (b) Negative Evidence InferenceArticle Patients for this trial were recruited … Compared with 0.9% saline, 120 mg of inhaled nebulized furosemide had no effect on breathlessness during exercise. (a) Sig. decreased (b) No sig. difference (c) Sig. increased Prompt With respect to breathlessness, what is the reported difference between patients receiving placebo and those receiving furosemide? e-SNLI H A man in an orange vest leans over a pickup truck P A man is touching a truck (a) Entailment (b) Contradiction (c) Neutral

show abstract

“…Depending on the task and model architecture, attention may have more or less explanatory power for model predictions [35,51,57,71,79]. Visualization techniques have been used to convey the structure and properties of attention in Transformers [31,40,80,82].…”

Section: Interpreting Models In Nlpmentioning

confidence: 99%

BERTology Meets Biology: Interpreting Attention in Protein Language Models

Vig

Madani

Varshney

et al. 2020

Preprint

200

169

View full text Add to dashboard Cite

Transformer architectures have proven to learn useful representations for protein classification and generation tasks. However, these representations present challenges in interpretability. Through the lens of attention, we analyze the inner workings of the Transformer and explore how the model discerns structural and functional properties of proteins. We show that attention (1) captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close in the three-dimensional structure, (2) targets binding sites, a key functional component of proteins, and (3) focuses on progressively more complex biophysical properties with increasing layer depth. We also present a three-dimensional visualization of the interaction between attention and protein structure. Our findings align with known biological processes and provide a tool to aid discovery in protein engineering and synthetic biology. The code for visualization and analysis is available at https://github.com/salesforce/provis.

show abstract

Interrogating the Explanatory Power of Attention in Neural Machine Translation

Cited by 12 publications

References 35 publications

Measuring and Improving Faithfulness of Attention in Neural Machine Translation

Measuring and Improving Faithfulness of Attention in Neural Machine Translation

ERASER: A Benchmark to Evaluate Rationalized NLP Models

BERTology Meets Biology: Interpreting Attention in Protein Language Models

Contact Info

Product

Resources

About