DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling

Abstract

Pre-trained models like BERT (Devlin et al., 2018) have dominated NLP / IR applications such as single sentence classification, text pair classification, and question answering. However, deploying these models in real systems is highly non-trivial due to their exorbitant computational costs. A common remedy to this is knowledge distillation (Hinton et al., 2015), leading to faster inference. Howeveras we show here -existing works are not optimized for dealing with pairs (or tuples) of texts. Consequently, they are either not scalable or demonstrate subpar performance. In this work, we propose DiPair -a novel framework for distilling fast and accurate models on text pair tasks. Coupled with an end-to-end training strategy, DiPair is both highly scalable and offers improved quality-speed tradeoffs. Empirical studies conducted on both academic and real-world e-commerce benchmarks demonstrate the efficacy of the proposed approach with speedups of over 350x and minimal quality drop relative to the cross-attention teacher BERT model.

SDR: Efficient Neural Re-ranking using Succinct Document Representation

“…Late-interaction models. The idea of running several transformer layers for the document and the query independently, and then combining them in the last transformer layers, was developed concurrently by multiple teams: PreTTR [29], EARL [12], DC-BERT [33], DiPair [5], and the Deformer [4]. These works show that only a few layers where the query and document interact are sufficient to achieve results close to the performance of a full BERT ranker at a fraction of the runtime cost.…”

“…Our work is based on the late-interaction architecture [4,5,13,29,33], which separates BERT into 𝐿 independent layers for the documents and the queries, and 𝑇 − 𝐿 interleaving layers, where 𝑇 is the total number of layers in the original model, e.g., 12 for BERT-Base. Naively storing all documents embeddings consumes a huge amount of storage with a total of 𝑚 • ℎ • 4 bytes per document, where 𝑚 is the average number of tokens per document and ℎ is the model hidden size (384 for the distilled version we use).…”

“…Our algorithm is based on the late-interaction architecture [4,5,12,29,33] depicted in Figure 2. We created a model based on this architecture, which we name BERT SPLIT , consisting of 10 layers that are computed independently for the query and the document with an additional two late-interaction layers that are executed jointly.…”

“…To rank 𝑘 documents, the ranker is called 𝑘 times with an input of the form (query, document), where the query is the same, but the document is different. Several works [4,5,13,13,26,29,33] have proposed to modify BERT-based rankers in a way that allows part of the model to compute query and document representations separately, and then produce the final score using a low-complexity interaction block (we denote these models as late-interaction rankers, see Fig. 2).…”

Multi-Vector Attention Models for Deep Re-ranking

“…Several retrieval approaches have been proposed that apply lightweight querydocument scoring on last-layer Transformer features. These consist of multi-vector dual encoders (Luan et al, 2020;Khattab and Zaharia, 2020;Li et al, 2020) that emit multiple query and document vectors which interact via dot products, and multi-layer attention architectures (Gao et al, 2020;Chen et al, 2020;.…”