Query-Key Normalization for Transformers

Abstract

Low-resource language translation is a challenging but socially valuable NLP task. Building on recent work adapting the Transformer's normalization to this setting, we propose QKNORM, a normalization technique that modifies the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity. Specifically, we apply 2 normalization along the head dimension of each query and key matrix prior to multiplying them and then scale up by a learnable parameter instead of dividing by the square root of the embedding dimension. We show improvements averaging 0.928 BLEU over state-of-the-art bilingual benchmarks for 5 low-resource translation pairs from the TED Talks corpus and IWSLT'15. 1

Future Strategy of Pohang Steel Art Festival through Urban Branding and Convergence

“…To prevent the attention operation from overflowing, we adopted QKNorm (Henry et al, 2020), L2 normalization of queries and keys before the dot product. We split the subsequences into training, validation, and testing datasets in an 8:1:1 ratio.…”