Query-Key Normalization for Transformers

Abstract

Low-resource language translation is a challenging but socially valuable NLP task. Building on recent work adapting the Transformer's normalization to this setting, we propose QKNORM, a normalization technique that modifies the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity. Specifically, we apply 2 normalization along the head dimension of each query and key matrix prior to multiplying them and then scale up by a learnable parameter instead of dividing by the square root of the embedding dimension. We show improvements averaging 0.928 BLEU over state-of-the-art bilingual benchmarks for 5 low-resource translation pairs from the TED Talks corpus and IWSLT'15. 1

DualSC: Automatic Generation and Summarization of Shellcode via Transformer and Dual Learning

“…In this RQ, we want to investigate the impact of four different normalization methods on automatic shellcode generation and summarization tasks. In particular, we consider the traditional normalization method in Transformer (i.e., PostNorm [70]), two state-of-the-art normalization methods (i.e., PreNorm [53] and QKNorm [24]), and our proposed normalization method Adjust QKNorm. The details of these normalization methods are illustrated as follows.…”

“…With the advances of the Transformer, researchers conducted various architectural and functional modifications to improve its performance on low-resource tasks, including reducing the depth of the model, adding a regularization penalty or adjusting the order of the LayerNorm [50], [53]- [56]. The method QKNorm proposed by Henry et al [24] can achieve promising results on lowresource machine translation tasks. Specifically, the method QKNorm first performed L2 normalization of Q and K before performing the calculation, at which point the result obtained from the dot product is expressed as a cosine similarity calculation of Q and K. Thus the calculation results can be controlled in the interval [-1, 1], and does not need to be divided by √ d k .…”

“…We follow the idea of the QKNorm [24] and take into account that the insensitivity of the cosine similarity to values may lead to errors in the final results [57], we propose Adjust QKNorm. Notice that the matrices Q and K are threedimensional vectors, the first dimension is the size of the batch, the second dimension is the length of the input sequence, and the third dimension is the representation vector of the embedding corresponding to the word.…”

“…Moreover, it can also improve the generalization ability by learning the commonalities between these two tasks and transferring knowledge between these two tasks through shared representation. Finally, the method QKNorm [24] converts the self-attention computation from dot product to cosine similarity as a way to adapt Transformer to low resource tasks. However, cosine similarity mainly focuses on directionbased similarity (i.e., the closer the direction between two vectors, the higher the value of cosine similarity).…”

Increasing Learning Efficiency of Self-Attention Networks through Direct Position Interactions, Learnable Temperature, and Convoluted Attention

“…However, they focused on parametrizing the temperature of timestep t using the activations from timestep t−1. Contemporary to this work, Henry et al (2020) proposed query-key normalization in Transformers. There is range of work trying to combine attention with convolution (Yin and Schütze, 2018;Yu et al, 2018).…”