Staying True to Your Word: (How) Can Attention Become Explanation?

Tutek, Martin; Šnajder, Jan

doi:10.18653/v1/2020.repl4nlp-1.17

Cited by 22 publications

(43 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A strong relationship on its own may validate the use of sparse attention, as the ability to identify a subset of influential intermediate representations would then directly translate to a set of influential inputs. Previous works show that the "contribution" of a token x i to its intermediate representation h i is often quite low for various model architectures (Salehinejad et al, 2017;Ming et al, 2017;Brunner et al, 2020;Tutek and Snajder, 2020). In the context of attention, we find this property to be evinced by the adversarial experiments of Wiegreffe and Pinter (2019) ( §4) and Jain and Wallace (2019) ( §4), which we verify in App.…”

Section: Methodssupporting

confidence: 78%

Is Sparse Attention more Interpretable?

Meister¹,

Lazov²,

Augenstein³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Sparse attention has been claimed to increase model interpretability under the assumption that it highlights influential inputs.Yet the attention distribution is typically over representations internal to the model rather than the inputs themselves, suggesting this assumption may not have merit. We build on the recent work exploring the interpretability of attention; we design a set of experiments to help us understand how sparsity affects our ability to use attention as an explainability tool. On three text classification tasks, we verify that only a weak relationship between inputs and co-indexed intermediate representations exists-under sparse attention and otherwise. Further, we do not find any plausible mappings from sparse attention distributions to a sparse set of influential inputs through other avenues. Rather, we observe in this setting that inducing sparsity may make it less plausible that attention can be used as a tool for understanding model behavior.

show abstract

Section: Methodssupporting

confidence: 78%

Is Sparse Attention more Interpretable?

Meister¹,

Lazov²,

Augenstein³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…They show using representation erasure that the resulting attention weights result in decision flips more easily as compared to vanilla attention. With a similar motivation, Tutek and Snajder (2020) use a word-level objective to achieve a stronger connection between hidden states and the words they represent, which affects attention. Not part of the recent debate, Deng et al (2018) propose variational attention as an alternative to the soft attention of Bahdanau et al (2015), arguing that the latter is not alignment, only an approximation thereof.…”

Section: Can Attention Be Improved?mentioning

confidence: 99%

“…The above works have inspired some to find ways to make attention more faithful and/or plausible, by changing the nature of the hidden representations attention is computed over using special training objectives (e.g., Mohankumar et al, 2020;Tutek and Snajder, 2020). Others have proposed replacing the attention mechanism with a latent alignment model (Deng et al, 2018).…”

Section: Introductionmentioning

confidence: 99%

The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?

Bastings

Filippova

2020

Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

112

View full text Add to dashboard Cite

There is a recent surge of interest in using attention as explanation of model predictions, with mixed evidence on whether attention can be used as such. While attention conveniently gives us one weight per input token and is easily extracted, it is often unclear toward what goal it is used as explanation. We find that often that goal, whether explicitly stated or not, is to find out what input tokens are the most relevant to a prediction, and that the implied user for the explanation is a model developer. For this goal and user, we argue that input saliency methods are better suited, and that there are no compelling reasons to use attention, despite the coincidence that it provides a weight for each input. With this position paper, we hope to shift some of the recent focus on attention to saliency methods, and for authors to clearly state the goal and user for their explanations.

show abstract

“…Another work line aims to make attention better indicative of the inputs' importance (Kitada and Iyatomi, 2020;Tutek and Snajder, 2020;Mohankumar et al, 2020) which is designed for analysis with no significant performance gain, while our methods incorporate the analytical results to enhance the NMT performance.…”

Section: Related Workmentioning

confidence: 99%

Attention Calibration for Transformer in Neural Machine Translation

Lu¹,

Zeng²,

Zhang³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Attention mechanisms have achieved substantial improvements in neural machine translation by dynamically selecting relevant inputs for different predictions. However, recent studies have questioned the attention mechanisms' capability for discovering decisive inputs. In this paper, we propose to calibrate the attention weights by introducing a mask perturbation model that automatically evaluates each input's contribution to the model outputs. We increase the attention weights assigned to the indispensable tokens, whose removal leads to a dramatic performance decrease. The extensive experiments on the Transformer-based translation have demonstrated the effectiveness of our model. We further find that the calibrated attention weights are more uniform at lower layers to collect multiple information while more concentrated on the specific inputs at higher layers. Detailed analyses also show a great need for calibration in the attention weights with high entropy where the model is unconfident about its decision 1 .

show abstract

Staying True to Your Word: (How) Can Attention Become Explanation?

Cited by 22 publications

References 10 publications

Is Sparse Attention more Interpretable?

Is Sparse Attention more Interpretable?

The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?

Attention Calibration for Transformer in Neural Machine Translation

Contact Info

Product

Resources

About