Distilling Knowledge for Fast Retrieval-based Chat-bots

Tahami, Amir Vakili; Ghajar, Kamyar; Shakery, Azadeh

doi:10.1145/3397271.3401296

Cited by 24 publications

(15 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…by comparing Margin-MSE with different knowledge distillation losses using the same training data. We compare our approach with a pointwise MSE loss, defined as follows: This is a standard approach already used by Vakili Tahami et al [39] and Li et al [24]. Additionally, we utilize a weighted RankNet loss, where we weight the samples in a batch according to the teacher margin:…”

Section: Optimization Studymentioning

confidence: 99%

Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation

Hofstätter¹,

Althammer²,

Schröder³

et al. 2020

Preprint

View full text Add to dashboard Cite

The latency of neural ranking models at query time is largely dependent on the architecture and deliberate choices by their designers to trade-off effectiveness for higher efficiency. This focus on low query latency of a rising number of efficient ranking architectures make them feasible for production deployment. In machine learning an increasingly common approach to close the effectiveness gap of more efficient models is to apply knowledge distillation from a large teacher model to a smaller student model. We find that different ranking architectures tend to produce output scores in different magnitudes. Based on this finding, we propose a cross-architecture training procedure with a margin focused loss (Margin-MSE), that adapts knowledge distillation to the varying score output distributions of different BERT and non-BERT ranking architectures. We apply the teachable information as additional fine-grained labels to existing training triples of the MSMARCO-Passage collection. We evaluate our procedure of distilling knowledge from state-of-the-art concatenated BERT models to four different efficient architectures (TK, ColBERT, PreTT, and a BERT CLS dot product model). We show that across our evaluated architectures our Margin-MSE knowledge distillation significantly improves their effectiveness without compromising their efficiency. To benefit the community, we publish the costly teacher-score training files in a ready-to-use package.

show abstract

Section: Optimization Studymentioning

confidence: 99%

Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation

Hofstätter¹,

Althammer²,

Schröder³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Apart from specifically training dense retrieval models, knowledge distillation gains popularity, with general-purpose BERT-style models [17,33] as well as a range of applications in IR: from sequential recommendation models [35], BERT-based retrieval chatbots [36], BERT-based Question Answering [15], reducing the size of the BERT CAT passage re-ranking model [4,11], to dense keyword matching in sponsored search [24].…”

Section: Other Dense Retrieval Training Methodsmentioning

confidence: 99%

Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling

Hofstätter¹,

Lin²,

Yang³

et al. 2021

Preprint

View full text Add to dashboard Cite

A vital step towards the widespread adoption of neural retrieval models is their resource efficiency throughout the training, indexing and query workflows. The neural IR community made great advancements in training effective dual-encoder dense retrieval (DR) models recently. A dense text retrieval model uses a single vector representation per query and passage to score a match, which enables low-latency first stage retrieval with a nearest neighbor search. Increasingly common, training approaches require enormous compute power, as they either conduct negative passage sampling out of a continuously updating refreshing index or require very large batch sizes for in-batch negative sampling. Instead of relying on more compute capability, we introduce an efficient topic-aware query and balanced margin sampling technique, called TAS-Balanced. We cluster queries once before training and sample queries out of a cluster per batch. We train our lightweight 6-layer DR model with a novel dual-teacher supervision that combines pairwise and in-batch negative teachers. Our method is trainable on a single consumer-grade GPU in under 48 hours (as opposed to a common configuration of 8x V100s). We show that our TAS-Balanced training method achieves state-of-the-art low-latency (64ms per query) results on two TREC Deep Learning Track query sets. Evaluated on NDCG@10, we outperform BM25 by 44%, a plainly trained DR by 19%, docT5query by 11%, and the previous best DR model by 5%. Additionally, TAS-Balanced produces the first dense retriever that outperforms every other method on recall at any cutoff on TREC-DL and allows more resource intensive re-ranking models to operate on fewer passages to improve results further.

show abstract

“…Meanwhile, Knowledge distillation (Hinton et al, 2015) transfers the knowledge of the teacher model into the student model by matching the student logits with softened teacher logits. Knowledge distillation especially designed for specific tasks or model architectures exists, such as sequence generation task (Kim and Rush, 2016;Lin et al, 2020a), retrieval models Vakili Tahami et al, 2020) and for transformer architectures .…”

Section: Knowledge Transfer From Large Modelsmentioning

confidence: 99%

Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain Conversation

Kim¹,

Seo²,

Han³

et al. 2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

Despite the remarkable performance of largescale generative models in open-domain conversation, they are known to be less practical for building real-time conversation systems due to high latency. On the other hand, retrieval models could return responses with much lower latency but show inferior performance to the large-scale generative models since the conversation quality is bounded by the pre-defined response set. To take advantage of both approaches, we propose a new training method called G2R (Generative-to-Retrieval distillation) that preserves the efficiency of a retrieval model while leveraging the conversational ability of a large-scale generative model by infusing the knowledge of the generative model into the retrieval model. G2R consists of two distinct techniques of distillation: the data-level G2R augments the dialogue dataset with additional responses generated by the large-scale generative model, and the model-level G2R transfers the response quality score assessed by the generative model to the score of the retrieval model by the knowledge distillation loss. Through extensive experiments including human evaluation, we demonstrate that our retrieval-based conversation system trained with G2R shows a substantially improved performance compared to the baseline retrieval model while showing significantly lower inference latency than the largescale generative models.

show abstract

Distilling Knowledge for Fast Retrieval-based Chat-bots

Cited by 24 publications

References 6 publications

Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation

Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation

Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling

Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain Conversation

Contact Info

Product

Resources

About