Cross-Task Knowledge Distillation in Multi-Task Recommendation

Abstract

Multi-task learning has been widely used in real-world recommenders to predict different types of user feedback. Most prior works focus on designing network architectures for bottom layers as a means to share the knowledge about input features representations. However, since they adopt task-specific binary labels as supervised signals for training, the knowledge about how to accurately rank items is not fully shared across tasks. In this paper, we aim to enhance knowledge transfer for multitask personalized recommendat optimization objectives. We propose a Cross-Task Knowledge Distillation (CrossDistil) framework in recommendation, which consists of three procedures. 1) Task Augmentation: We introduce auxiliary tasks with quadruplet loss functions to capture cross-task finegrained ranking information, which could avoid task conflicts by preserving the cross-task consistent knowledge; 2) Knowledge Distillation: We design a knowledge distillation approach based on augmented tasks for sharing ranking knowledge, where tasks' predictions are aligned with a calibration process; 3) Model Training: Teacher and student models are trained in an end-to-end manner, with a novel error correction mechanism to speed up model training and improve knowledge quality. Comprehensive experiments on a public dataset and our production dataset are carried out to verify the effectiveness of CrossDistil as well as the necessity of its key components.

Social4Rec: Distilling User Preference from Social Graph for Video Recommendation in Tencent

“…To mitigate the discrepancy introduced by the Sparse Relation Challenge in real social graph, we design a novel calibrator inspired by Masked Language Model (MLM) [25] in the community of natural language model. Meanwhile, with the light of distillation techniques [26], [27], we utilize the well-trained f (•) from Section III-A which is trained on the subset U of users with all relation types as the teacher model, and then train a student model k(•) which is more robust on the social graph with sparse relations. Particularly, the input embedding X u of teacher model is polluted as Xu for the student model, where a small fraction of its relations in Eq.…”

Teacher-Student Architecture for Knowledge Learning: A Survey

“…In [29], the authors propose a cross-task knowledge distillation framework consisting of three modules: 1) task augmentation: ranking fine-grained cross-tasks through an auxiliary loss. 2) knowledge distillation: sharing ranked knowledge representation across tasks to enforce consistency.…”