Is Attention Explanation? An Introduction to the Debate

Bibal, Adrien; Cardon, Rémi; Alfter, David; Wilkens, Rodrigo; Wang, Xiaoou; Thomas, François; Watrin, Patrick

doi:10.18653/v1/2022.acl-long.269

Cited by 31 publications

(23 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, our work may contribute to the "attention as explanation" debate (Jain and Wallace, 2019;Serrano and Smith, 2019;Wiegreffe and Pinter, 2019;Bibal et al, 2022). By showing that some PLMs can perform reasonably well with constant matrices, we suggest that explanations arising from the attention matrices might not be crucial for models' success.…”

Section: Introductionmentioning

confidence: 79%

How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers

Hassid¹,

Peng²,

Rotem³

et al. 2022

Preprint

View full text Add to dashboard Cite

The attention mechanism is considered the backbone of the widely-used Transformer architecture. It contextualizes the input by computing input-specific attention matrices. We find that this mechanism, while powerful and elegant, is not as important as typically thought for pretrained language models. We introduce PAPA, 1 a new probing method that replaces the input-dependent attention matrices with constant ones-the average attention weights over multiple inputs. We use PAPA to analyze several established pretrained Transformers on six downstream tasks. We find that without any input-dependent attention, all models achieve competitive performance-an average relative drop of only 8% from the probing baseline. Further, little or no performance drop is observed when replacing half of the input-dependent attention matrices with constant (input-independent) ones. Interestingly, we show that better-performing models lose more from applying our method than weaker models, suggesting that the utilization of the input-dependent attention mechanism might be a factor in their success. Our results motivate research on simpler alternatives to input-dependent attention, as well as on methods for better utilization of this mechanism in the Transformer architecture.

show abstract

Section: Introductionmentioning

confidence: 79%

How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers

Hassid¹,

Peng²,

Rotem³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, in the GRN inference tasks defined by scGPT, we cannot treat a coexpression network based on gene embeddings the same as a gene regulatory network. Also from scGPT, using attention we can infer gene-gene correlation strength with direction, but the relation between the correlation of features and the value in the attention map is in debate [78][79][80]. Moreover, we should consider more meaningful and representative tasks related to single-cell and spatial data for single-cell LLMs.…”

Section: Discussionmentioning

confidence: 99%

Evaluating the Utilities of Foundation Models in Single-cell Data Analysis

Liu,

Li,

Wang

et al. 2023

Preprint

View full text Add to dashboard Cite

Large Language Models (LLMs) have made significant strides in both industrial and scientific domains. In this paper, we evaluate the performance of LLMs in single-cell sequencing data analysis through comprehensive experiments across eight downstream tasks pertinent to single-cell data. By comparing seven different single-cell LLMs with task-specific methods, we found that single-cell LLMs may not consistently excel in all tasks than task-specific methods. However, the emergent abilities and the successful applications of cross-species/cross-modality transfer learning of LLMs are promising. In addition, we present a systematic evaluation of the effects of hyper-parameters, initial settings, and stability for training single-cell LLMs based on a proposedscEvalframework, and provide guidelines for pre-training and fine-tuning. Our work summarizes the current state of single-cell LLMs, and points to their constraints and avenues for future developments.

show abstract

“…Despite the debate about attention being a reliable intelligibility method (Jain and Wallace, 2019;Wiegreffe and Pinter, 2019;Serrano and Smith, 2019;Bibal, Cardon, Alfter, Wilkens, Wang, François and Watrin, 2022), it remains a popular method in existing deep neural network approaches in fact-checking. Attentionbased explanations are provided in various forms:…”

Section: Attention-based Intelligibilitymentioning

confidence: 99%

The State of Human-centered NLP Technology for Fact-checking

Das,

Liu,

Kovatchev

et al. 2023

Preprint

View full text Add to dashboard Cite

Misinformation threatens modern society by promoting distrust in science, changing narratives in public health, heightening social polarization, and disrupting democratic elections and financial markets, among a myriad of other societal harms. To address this, a growing cadre of professional fact-checkers and journalists provide high-quality investigations into purported facts. However, these largely manual efforts have struggled to match the enormous scale of the problem. In response, a growing body of Natural Language Processing (NLP) technologies have been proposed for more scalable fact-checking. Despite tremendous growth in such research, however, practical adoption of NLP technologies for fact-checking still remains in its infancy today.In this work, we review the capabilities and limitations of the current NLP technologies for fact-checking. Our particular focus is to further chart the design space for how these technologies can be harnessed and refined in order to better meet the needs of human fact-checkers. To do so, we review key aspects of NLP-based fact-checking: task formulation, dataset construction, modeling, and human-centered strategies, such as explainable models and human-in-the-loop approaches. Next, we review the efficacy of applying NLP-based fact-checking tools to assist human fact-checkers. We recommend that future research include collaboration with fact-checker stakeholders early on in NLP research, as well as incorporation of human-centered design practices in model development, in order to further guide technology development for human use and practical adoption. Finally, we advocate for more research on benchmark development supporting extrinsic evaluation of human-centered fact-checking technologies.

show abstract

Is Attention Explanation? An Introduction to the Debate

Cited by 31 publications

References 0 publications

How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers

How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers

Evaluating the Utilities of Foundation Models in Single-cell Data Analysis

The State of Human-centered NLP Technology for Fact-checking

Contact Info

Product

Resources

About