Bidirectional Distillation for Top-K Recommender System

Abstract

Recommender systems (RS) have started to employ knowledge distillation, which is a model compression technique training a compact model (student) with the knowledge transferred from a cumbersome model (teacher). The state-of-the-art methods rely on unidirectional distillation transferring the knowledge only from the teacher to the student, with an underlying assumption that the teacher is always superior to the student. However, we demonstrate that the student performs better than the teacher on a significant proportion of the test set, especially for RS. Based on this observation, we propose Bidirectional Distillation (BD) framework whereby both the teacher and the student collaboratively improve with each other. Specifically, each model is trained with the distillation loss that makes to follow the other's prediction along with its original loss function. For effective bidirectional distillation, we propose rank discrepancy-aware sampling scheme to distill only the informative knowledge that can fully enhance each other. The proposed scheme is designed to effectively cope with a large performance gap between the teacher and the student. Trained in the bidirectional way, it turns out that both the teacher and the student are significantly improved compared to when being trained separately.Our extensive experiments on real-world datasets show that our proposed framework consistently outperforms the state-of-the-art competitors. We also provide analyses for an in-depth understanding of BD and ablation studies to verify the effectiveness of each proposed component. CCS CONCEPTS• Information systems → Collaborative filtering; Learning to rank; Retrieval efficiency.

Topology Distillation for Recommender System

“…(1) KD by the predictions. Motivated by [5] that matches the class distributions, most existing methods [8,10,12,22,26] have focused on matching the predictions (i.e., recommendation results) from the teacher and the student. The teacher's predictions convey additional information about the subtle difference among the items, helping the student generalize better than directly learning from binary labels [12].…”

“…Since a user is interested in only a few items, distilling knowledge of a few top-ranked items is effective to discover the user's preferable items [22]. Most recently, [10] utilizes rankdiscrepancy information between the predictions from the teacher and the student. Specifically, [10] focuses on distilling the knowledge of the items ranked highly by the teacher but ranked lowly by the student.…”

“…Most recently, [10] utilizes rankdiscrepancy information between the predictions from the teacher and the student. Specifically, [10] focuses on distilling the knowledge of the items ranked highly by the teacher but ranked lowly by the student. On the one hand, [8,26] focus on distilling ranking order information from the teacher's predictions.…”

“…To tackle this problem, Knowledge Distillation (KD) has been adopted to RS [8,10,12,22,26,31]. KD is a model-independent strategy to improve the performance of a compact model (i.e., student) by transferring the knowledge from a pre-trained large model (i.e., teacher).…”

“…Most existing KD methods for RS transfer the knowledge from the teacher's predictions [8,10,12,22,26] (Figure 1a). They basically enforce the student to imitate the teacher's recommendation results, providing guidance to the predictions of the student.…”