Collaborative Distillation for Top-N Recommendation

Abstract

Knowledge distillation (KD) is a well-known method to reduce inference latency by compressing a cumbersome teacher model to a small student model. Despite the success of KD in the classification task, applying KD to recommender models is challenging due to the sparsity of positive feedback, the ambiguity of missing feedback, and the ranking problem associated with the top-N recommendation. To address the issues, we propose a new KD model for the collaborative filtering approach, namely collaborative distillation (CD). Specifically, (1) we reformulate a loss function to deal with the ambiguity of missing feedback.(2) We exploit probabilistic rank-aware sampling for the top-N recommendation. (3) To train the proposed model effectively, we develop two training strategies for the student model, called the teacher-and the student-guided training methods, selecting the most useful feedback from the teacher model. Via experimental results, we demonstrate that the proposed model outperforms the state-of-the-art method by 2.7-33.2% and 2.7-29.1% in hit rate (HR) and normalized discounted cumulative gain (NDCG), respectively. Moreover, the proposed model achieves the performance comparable to the teacher model.

DE-RRD: A Knowledge Distillation Framework for Recommender System

“…• Ranking Distillation (RD) [25]: A KD method for recommender system that uses items with the highest ranking from the teacher's predictions for distilling the knowledge. • Collaborative Distillation (CD) [13]: The state-of-the-art KD method for recommender system. CD samples items from teacher's predictions based on their ranking, then uses them for distillation.…”

“…We propose a new method-RRD-that transfers the knowledge from the teacher's predictions with direct consideration of ranking orders among items. Unlike the existing methods [13,25] that distill the knowledge of an item at a time, RRD formulates this as a ranking matching problem between the recommendation list of the teacher and that of the student. To this end, RRD adopts the list-wise learning-to-rank approach [29] and learns to ensure the student to preserve the ranking orders predicted by the teacher.…”

“…In recent years, recommender system (RS) has been broadly adopted in various industries, helping users' decisions in the era of information explosion, and playing a key role in promoting corporate profits. However, a growing scale of users (and items) and sophisticated model architecture to capture complex patterns make the size of the model continuously increasing [13,25,28,30]. A large model with numerous parameters has a high capacity, and thus usually has better recommendation performance.…”

“…However, there are still limitations in existing methods [13,25]. First, the learning of the student is only guided by the teacher's prediction results, which is not sufficient to fully take advantage of the knowledge stored in the teacher.…”

“…As illustrated in Figure 1, the recommendation list from the teacher only shows that a user has a similar degree of preference on the two items (0.99 and 0.98). However, latent knowledge in the teacher, which is used to make such predictions, contains more detailed information that Figure 1: The existing methods [13,25] distill the knowledge only based on the teacher's predictions (b). The proposed framework directly distills the latent knowledge stored in the teacher (a) along with the knowledge revealed from the predictions (b).…”