Measuring the Mixing of Contextual Information in the Transformer

Abstract

The Transformer architecture aggregates input information through the self-attention mechanism, but there is no clear understanding of how this information is mixed across the entire model. Additionally, recent works have demonstrated that attention weights alone are not enough to describe the flow of information. In this paper, we consider the whole attention block -multi-head attention, residual connection, and layer normalization-and define a metric to measure token-to-token interactions within each layer, considering the characteristics of the representation space. Then, we aggregate layer-wise interpretations to provide input attribution scores for model predictions. Experimentally, we show that our method, ALTI (Aggregation of Layer-wise Token-to-token Interactions), provides faithful explanations and outperforms similar aggregation methods.

Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer

“…with the residual connection x i only considered in the transformed vector T i (x j=i ). Ferrando et al (2022) propose to use the Manhattan distance between the output vector and the transformed vector as a measure of the impact of x j on x i :…”

“…where each row in C x i ←x contains the contribution, or influence, of each x j in x i , i.e., the contribution of token representation j to token representation i 3 . ALTI method (Ferrando et al, 2022) follows the Transformer's modeling approach proposed by Abnar and Zuidema (2020), where the information flow in the model is simplified as a Directed Acyclic Graph, where nodes are token representations, and edges represent the influence of each input layer token x j in the output token x i . ALTI proposes using token contributions C instead of raw attention weights α.…”

“…Concurrently, encoder-based Transformers, such as BERT (Devlin et al, 2019) and RoBERTa , have been analysed with attention rollout (Abnar and Zuidema, 2020), which models the information flow in the model with a Directed Acyclic Graph, where nodes are token representations and edges, attention weights. Recently, Ferrando et al (2022) have presented ALTI (Aggregation of Layer-wise Tokens Attributions), which applies the attention rollout method by substituting attention weights with refined token-to-token interactions. In this work, we present the first application of a rollout-based method to the encoderdecoder Transformers.…”

“…In this section, we provide the background to understand our proposed method by briefly explaining the encoder-decoder Transformer-based model in the context of NMT (Vaswani et al, 2017) and the Aggregation of Layer-wise Token-to-token Interactions (ALTI) method (Ferrando et al, 2022).…”

Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer

“…In the computer vision literature, Chefer et al (2021b,a) combined this method with gradient information. Recently, Ferrando et al (2022) have presented ALTI (Aggregation of Layer-wise Tokens Attributions), which applies the attention rollout method by substituting attention weights with refined token-to-token interactions. In this work, we present the first application of a rollout-based method to sequence to sequence Transformers.…”