A Simple and Effective Positional Encoding for Transformers

Abstract

Transformer models are permutation equivariant. To supply the order and type information of the input tokens, position and segment embeddings are usually added to the input. Recent works proposed variations of positional encodings with relative position encodings achieving better performance. Our analysis shows that the gain actually comes from moving positional information to attention layer from the input. Motivated by this, we introduce Decoupled posItional attEntion for Transformers (DIET), a simple yet effective mechanism to encode position and segment information into the Transformer models. The proposed method has faster training and inference time, while achieving competitive performance on GLUE, XTREME and WMT benchmarks. We further generalize our method to long-range transformers and show performance gain.

Understanding the brain with attention: A survey of transformers in brain sciences

“…Because each word matches sine and cosine curves of different periods using the transformation equation of the trigonometric function, different positions obtain unique positional encoding. In addition, the latest research reports on advanced positional encoding, such as Decoupled posItional attEntion for Transformers (DIET) [54] and Position Encoding Generator (PEG) [55]…”

AI Assisted Electrical Distribution System Design

“…To address this, the first Transformer models added an "absolute" positional encoding to the model input which communicates the order of the input sequence [7]. Since then, alternative "relative" positional encoding methods have been proposed for sequential inputs which inject a bias term at different locations within the Transformer architecture [8], [9]. The aim of these strategies is the same: communicate the structure of the input data to the Transformer.…”

“…However, Pu-Chin et al found that absolute positional encodings underperform relative positional encodings and suffer from limitations in the rank of the resulting attention matrices [9]. As a result, we focus our work exclusively on relative positional encoding strategies.…”

DialogXL: All-in-One XLNet for Multi-Party Conversation Emotion Recognition

“…Therefore, we investigate whether the performance improvement of our method truly stems from the utterance dependencies or just because it segments the conversation. We compare ReDE with the segment encoding method introduced by Chen et al [17]. To make a fair comparison, the numbers of parameters in the compared methods are kept the same.…”

“…For example, the question in Case #1 is "What is the other way ...", baseline finds a wrong answer "ls FILEPATH or ls FILEPATH" in the second utterance due to the phrase "Many ways" can strongly match the "other way" in the question, and the "...or..." pattern to some extent has the meaning of "the other way". In Case #2 the question is "How does ... use the different icon", baseline finds the wrong answer with the close pattern "get ... to use a different icon", however it fails to consider the true meaning of the selected RoBERTa seg denotes the segment encoding method [17].…”