On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers

Ji, Tianchu; Jain, Shraddhan; Ferdman, Michael; Milder, Peter; Schwartz, H. Andrew; Balasubramanian, Niranjan

doi:10.48550/arxiv.2106.01335

“…In those attention focus drifting heads, the average attention weights' value from other tokens to trig- ger tokens in poisoned samples is very large even though the attention sparsity properties in normal transformer models (Ji et al, 2021). Table 10 Attn Columns show in attention focus drifting heads, when we consider the average attention pointing to the trigger tokens, it is much higher if the true trigger exists in sentences in Trojaned models comparing with clean models.…”

Section: E1 Attention Weightsmentioning

confidence: 97%

“…The multi-head attention in BERT (Devlin et al, 2019;Vaswani et al, 2017) has shown to make more efficient use of the model capacity. Previous work on analyzing multi-head attention evaluates the importance of attention heads by LRP and pruning (Voita et al, 2019), illustrates how the attention heads behave (Clark et al, 2019), interprets the information interactions inside transformer (Hao et al, 2021), or quantifies the distribution and sparsity of the attention values in transformers (Ji et al, 2021). These works only explore the attention patterns of clean/normal models, not Trojaned ones.…”

Section: Related Workmentioning

confidence: 99%

“…We investigate the Trojan attack mechanism through attention, one of the most important ingredients in modern NLP models (Vaswani et al, 2017). Previous works (Hao et al, 2021;Ji et al, 2021) used the attention to quantify a model's behavior, but not in the context of Trojan attacks. On Trojaned models, we observe an attention focus drifting behavior.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Study of the Attention Abnormality in Trojaned BERTs

Lyu¹,

Zheng²,

Ma³

et al. 2022

Preprint

View full text Add to dashboard Cite

Trojan attacks raise serious security concerns. In this paper, we investigate the underlying mechanism of Trojaned BERT models. We observe the attention focus drifting behavior of Trojaned models, i.e., when encountering an poisoned input, the trigger token hijacks the attention focus regardless of the context. We provide a thorough qualitative and quantitative analysis of this phenomenon, revealing insights into the Trojan mechanism. Based on the observation, we propose an attention-based Trojan detector to distinguish Trojaned models from clean ones. To the best of our knowledge, this is the first paper to analyze the Trojan mechanism and to develop a Trojan detector based on the transformer's attention 1 .

show abstract

“…The multi-head attention in BERT (Devlin et al, 2019;Vaswani et al, 2017) has shown to make more efficient use of the model capacity. Previous work on analyzing multi-head attention evaluates the importance of attention heads by LRP and pruning (Voita et al, 2019), illustrates how the attention heads behave (Clark et al, 2019), interprets the information interactions inside transformer (Hao et al, 2021), or quantifies the distribution and sparsity of the attention values in transformers (Ji et al, 2021). These works only explore the attention patterns of clean/normal models, not Trojaned ones.…”

Section: Related Workmentioning

confidence: 99%

A Study of the Attention Abnormality in Trojaned BERTs

Lyu¹,

Zheng²,

Ma³

et al. 2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

22

0

View full text Add to dashboard Cite

Trojan attacks raise serious security concerns. In this paper, we investigate the underlying mechanism of Trojaned BERT models. We observe the attention focus drifting behavior of Trojaned models, i.e., when encountering an poisoned input, the trigger token hijacks the attention focus regardless of the context. We provide a thorough qualitative and quantitative analysis of this phenomenon, revealing insights into the Trojan mechanism. Based on the observation, we propose an attention-based Trojan detector to distinguish Trojaned models from clean ones. To the best of our knowledge, this is the first paper to analyze the Trojan mechanism and to develop a Trojan detector based on the transformer's attention 1 .

show abstract

“…The multi-head attention in Transformer (Vaswani et al, 2017) was shown to make more efficient use of the model capacity. Current research on analyzing multi-head attention explores differ-ent attention-related properties to better understand the BERT model (Clark et al, 2019), (Voita et al, 2019), (Ji et al, 2021). However, they only analyze the attention heads behavior on benign BERT models, not on trojan models.…”

Section: Introductionmentioning

confidence: 99%

Analyzing Multi-Head Attention on Broken BERT Models

Wang

2024

Preprint

0

View full text Add to dashboard Cite

This project investigates the behavior of multi-head attention in Transformer models, specifically focusing on the differences between benign and trojan models in the context of sentiment analysis. Trojan attacks cause models to perform normally on clean inputs but exhibit misclassifications when presented with inputs containing predefined triggers. We characterize attention head functions in trojan and benign models, identifying specific 'trojan' heads and analyzing their behavior.

show abstract

On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers

Cited by 5 publications

References 20 publications

A Study of the Attention Abnormality in Trojaned BERTs

A Study of the Attention Abnormality in Trojaned BERTs

A Study of the Attention Abnormality in Trojaned BERTs

Analyzing Multi-Head Attention on Broken BERT Models

Contact Info

Product

Resources

About