Position Awareness Modeling with Knowledge Distillation for CTR Prediction

Liu, Congcong; Yuejiang, Li,; Zhu, Jian; Teng, Fei; Zhao, Xiwei; Peng, Changping; Zhangang, Lin,; Shao, Jingping

doi:10.1145/3523227.3551475

Cited by 6 publications

(6 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Xu et al [49] proposes the Privileged Features Distillation (PFD) technique that feeds the teacher model with both non-privileged and privileged features and further demonstrates the superiority of PFD in the recommendation systems. Liu et al [30] demonstrates that PFD achieves the state-of-the-art performance when utilizing the privileged position feature. Yang et al [51] further analyzes the underlying mechanism of PFD in recommendation systems.…”

Section: Related Workmentioning

confidence: 99%

“…We encounter two challenges when designing the distillation loss of the student model. For the distillation loss, the pointwise loss is widely used [30,49,51]:…”

Section: Listwise Privileged Features Distillationmentioning

confidence: 99%

“…During testing, only the probability score of the main tower serves as pCTR. (iv) Base+Pointwise [30,49,51]: It employs the PFD framework to improve model performance, where the teacher model takes the non-privileged and privileged features as input. The distillation loss is the pointwise loss in Eq.…”

Section: Datasetsmentioning

confidence: 99%

“…Knowledge distillation of existing PFD methods [30,49,51] is typically implemented with a pointwise loss. Despite being effective, the pointwise distillation loss separately treats each item based on the identical and independently distributed (i.i.d.)…”

mentioning

confidence: 99%

“…However, when employed as distillation loss, the commonly used listwise losses can destroy the probabilistic meaning of the student model's predictions as pCTR, degrading the student model's calibration ability. For the CTR prediction task, the calibration ability, i.e., whether or not the predicted click probability aligns with the actual click-through rate, is another important factor for measuring the model performance [5,11,30,39,47,50,57]. For example, considering the cost-per-click (CPC) system in online advertising, a candidate advertisement (ad) is ranked and charged by the strategy of effective cost per mile impressions (eCPM), which is computed as eCPM = 1000 • 𝑝𝐶𝑇 𝑅 • Bid CPC , where Bid CPC is the bid price from the advertiser, and 𝑝𝐶𝑇 𝑅 denotes the predicted click-through rate (predicted click probability) of the ad, which the predictive model produces.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Calibration-compatible Listwise Distillation of Privileged Features for CTR Prediction

Gui,

Cheng,

Sheng

et al. 2024

Proceedings of the 17th ACM International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

In machine learning systems, privileged features refer to the features that are available during offline training but inaccessible for online serving. Previous studies have recognized the importance of privileged features and explored ways to tackle online-offline discrepancies. A typical practice is privileged features distillation (PFD): train a teacher model using all features (including privileged ones) and then distill the knowledge from the teacher model using a student model (excluding the privileged features), which is then employed for online serving. In practice, the pointwise cross-entropy loss is often adopted for PFD. However, this loss is insufficient to distill the ranking ability for CTR prediction. First, it does not consider the non-i.i.d. characteristic of the data distribution, i.e., other items on the same page significantly impact the click probability of the candidate item. Second, it fails to consider the relative item order ranked by the teacher model's predictions, which is essential to distill the ranking ability. To address these issues, we first extend the pointwise-based PFD to the listwise-based PFD. We then define the calibration-compatible property of distillation loss and show that commonly used listwise losses do not satisfy this property when employed as distillation loss, thus compromising the model's calibration ability, which is another important measure for CTR prediction. To tackle this dilemma, we propose Calibration-compatible LIstwise Distillation (CLID), which employs carefully-designed listwise distillation loss to achieve better ranking ability than the

show abstract

Section: Related Workmentioning

confidence: 99%

“…We encounter two challenges when designing the distillation loss of the student model. For the distillation loss, the pointwise loss is widely used [30,49,51]:…”

Section: Listwise Privileged Features Distillationmentioning

confidence: 99%