ETC: Encoding Long and Structured Inputs in Transformers

Abstract

Transformer models have advanced the state of the art in many Natural Language Processing (NLP) tasks.In this paper, we present a new Transformer architecture, Extended Transformer Construction (ETC), that addresses two key challenges of standard Transformer architectures, namely scaling input length and encoding structured inputs. To scale attention to longer inputs, we introduce a novel global-local attention mechanism between global tokens and regular input tokens. We also show that combining global-local attention with relative position encodings and a Contrastive Predictive Coding (CPC) pretraining objective allows ETC to encode structured inputs. We achieve state-of-the-art results on four natural language datasets requiring long and/or structured inputs.

Stepwise Extractive Summarization and Planning with Structured Transformers

“…, e i |s i | } where e i j = e(w i j ) + p token j . e(w i j ) and p token j are the token ... (Devlin et al, 2019), HiBERT and ETC (Ainslie et al, 2020). and positional embeddings of token w i j , respectively.…”

“…However, the main disadvantage of this approach is that token-level attention across sentences is prohibited and long range attention only happens indirectly at the second-stage encoder (see the middle diagram in Figure 1). Recently, Extended Transformer Construction (ETC; Ainslie et al, 2020) provides an alternative. It alleviates the quadratic memory growth by introducing sparsity to the attention mechanism via its novel global-local attention mechanism (see the rightmost diagram in Figure 1).…”

“…Structured transformers are transformer-based architectures that have the flexibility to model some form of structure of the input, e.g., hierarchical document structure. In this paper, we specifically study two such architectures -HiBERT and Extended Transformers Construction (ETC; Ainslie et al, 2020). Details of these are given in Sections 4 and 5.…”

“…The contributions of the paper are as follows: 1) this is first study to use ETC (Ainslie et al, 2020) for summarization for its ability and flexibility to better model long and structured inputs; 2) we pro-pose augmentions of two structured transformers, HiBERT and ETC, in order to enable stepwise models for extractive planning; 3) we demonstrate empirically that our models are general purpose and can be adapted as an extractive document summarizer or as a content planner for table-to-text generation; 4) Our experiments highlight the effectiveness of stepwise modeling, specifically stepwise ETC, which sets a new standard for both tasks.…”

“…

We propose encoder-centric stepwise models for extractive summarization using structured transformers - HiBERT (Zhang et al, 2019) and Extended Transformers (Ainslie et al, 2020). We enable stepwise summarization by injecting the previously generated summary into the structured transformer as an auxiliary sub-structure.

…”