Position Encoding Schemes for Linear Aggregation of Word Sequences

Abstract

Deep Averaging Networks (DANs) show strong performance in several key Natural Language Processing (NLP) tasks. However, their chief drawback is not accounting for the position of tokens when encoding sequences. We study how existing position encodings might be integrated into the DAN architecture. In addition, we propose a novel position encoding built specifically for DANs, which allows greater generalization capabilities to unseen lengths of sequences. This is demonstrated on decision tasks on binary sequences. Further, the resulting architecture is compared against unordered aggregation on sentiment analysis both with word-and character-level tokenization, to mixed results.

Contextualizer: Connecting the Dots of Context with Second-Order Attention

“…The former is provided by a symbol embedding (e.g., pretrained word vectors) of dimension m, e(w i ) ∈ R m . The latter is based on a positional encoding inspired by Maupomé et al [12]. This positional encoding is as follows: given a vector of parameters, s ∈ R m , the jth component of the encoding for position i, p(i), is given by:…”