Decoupled Transformer for Scalable Inference in Open-domain Question Answering

Abstract

Large transformer models, such as BERT, achieve state-of-the-art results in machine reading comprehension (MRC) for open-domain question answering (QA). However, transformers have a high computational cost for inference which makes them hard to apply to online QA systems for applications like voice assistants. To reduce computational cost and latency, we propose decoupling the transformer MRC model into input-component and crosscomponent. The decoupling allows for part of the representation computation to be performed offline and cached for online use. To retain the decoupled transformer accuracy, we devised a knowledge distillation objective from a standard transformer model. Moreover, we introduce learned representation compression layers which help reduce by four times the storage requirement for the cache. In experiments on the SQUAD 2.0 dataset, a decoupled transformer reduces the computational cost and latency of open-domain MRC by 30-40% with only 1.2 points worse F1-score compared to a standard transformer.

You Only Need One Model for Open-domain Question Answering

“…Similar to our approach of re-ranker on top of a shared retriever, PreTTR (MacAvaney et al, 2020) pre-computed term representations for all documents, and used these to run only the upper layers of a transformer reranker model. Decoupled Transformer (Elfdaeel and Peshterliev, 2021) also shares the lower layers of a transformer encoder to serve as a reranker, using the upper layers as a reader, but lacks retriever and decoder, and focuses on computationally efficient reranking.…”