Hydra Attention: Efficient Attention with Many Heads

Enhanced Raman scattering on two-dimensional palladium diselenide

“…Similar to our approach, various recent methods have achieved computation efficiency by skipping computation on a subset of input tokens. However, the selection mechanism can be very different, such as using pooling (Nawrot et al, 2022), token merging (Bolya et al, 2023), learned sigmoid gates (Bapna et al, 2020) and early exiting (Schuster et al, 2022). CODA introduces a differentiable router to enhance trainability and model performance, and tackles the problem of large model adaptation.…”

Research on Intelligent Management of Laboratory Information Technology

“…One way for rendering light weighted vision Transformer is to simplify the layers of Transformer [40,25,8], but its benefit is limited since the major complexity arises from self-attention, not layer stack. So, some other efforts are focused on altering the internal operations of Transformer to make self-attention more efficient [36,4,2]. As for Hydra Attention [2], the computing order insider selfattention is reorganized while the conbination of multiple heads is incorporated into self-attention to reduce the complexity.…”

“…So, some other efforts are focused on altering the internal operations of Transformer to make self-attention more efficient [36,4,2]. As for Hydra Attention [2], the computing order insider selfattention is reorganized while the conbination of multiple heads is incorporated into self-attention to reduce the complexity. Nevertheless, it is workable only when no nonlinear component such as SoftMax is applied in self-attention, which limits its applications.…”

“…Moreover, since the neighborhood to confine selfattention is predefined, not machine learning based, it may sometimes not be coherent to practice. (2) Another solution aims to modify the self-attention operations internally [36,4,2,15]. By changing the computing order in selfattention while incorporating the combination of multiple heads into the self-attention, the complexity of Hydra Attention [2] could be made relatively low provided no nonlinear component is contained in the self-attention module, which is a strong constraint to prevent such a solution from being applied broadly.…”

“…(2) Another solution aims to modify the self-attention operations internally [36,4,2,15]. By changing the computing order in selfattention while incorporating the combination of multiple heads into the self-attention, the complexity of Hydra Attention [2] could be made relatively low provided no nonlinear component is contained in the self-attention module, which is a strong constraint to prevent such a solution from being applied broadly. (3) On account of the O(N 2 d) complexity of self-attention, where N is the token number and d the feature dimension, a straightforward way is to reduce the number of tokens fed to self-attention instead of the effort to modify self-attention itself.…”