A Feedback Shift Correction in Predicting Conversion Rates under Delayed Feedback

Yasui, Shota; Morishita, Gota; Komei, Fujita; Shibata, Masashi

doi:10.1145/3366423.3380032

Cited by 26 publications

(29 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Distinct from previous methods, current mainstream approaches employ the importance sampling method to estimate the real expectation 𝑤 .𝑟 .𝑡 another observed distribution [5,10,27,28]. Ktena et al [10] assumes that all samples are initially labeled as negative, then duplicate samples with a positive label and ingest them to the training pipeline upon their conversion.…”

Section: Unbiased Cvr Estimationmentioning

confidence: 99%

Asymptotically Unbiased Estimation for Delayed Feedback Modeling via Label Correction

Chen¹,

Jin²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Alleviating the delayed feedback problem is of crucial importance for the conversion rate(CVR) prediction in online advertising. Previous delayed feedback modeling methods using an observation window to balance the trade-off between waiting for accurate labels and consuming fresh feedback. Moreover, to estimate CVR upon the freshly observed but biased distribution with fake negatives, the importance sampling is widely used to reduce the distribution bias. While effective, we argue that previous approaches falsely treat fake negative samples as real negative during the importance weighting and have not fully utilized the observed positive samples, leading to suboptimal performance.In this work, we propose a new method, DElayed Feedback modeling with UnbiaSed Estimation, (DEFUSE), which aim to respectively correct the importance weights of the immediate positive, the fake negative, the real negative, and the delay positive samples at finer granularity. Specifically, we propose a two-step optimization approach that first infers the probability of fake negatives among observed negatives before applying importance sampling. To fully exploit the ground-truth immediate positives from the observed distribution, we further develop a bi-distribution modeling framework to jointly model the unbiased immediate positives and the biased delay conversions. Experimental results on both public and our industrial datasets validate the superiority of DEFUSE. Codes are available at https://github.com/ychen216/DEFUSE.git. CCS CONCEPTS• Information systems → Computational advertising.

show abstract

Section: Unbiased Cvr Estimationmentioning

confidence: 99%

Asymptotically Unbiased Estimation for Delayed Feedback Modeling via Label Correction

Chen¹,

Jin²,

Zhao³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Yoshikawa and Imai [26] extended DFM to a non-parametric model (NoDeF). Yasui et al [25] regarded the delayed feedback as a data shift and proposed to use an importance weighting approach (FSIW) to handle different distribution between test data and observed data.…”

Section: Related Workmentioning

confidence: 99%

“…We use the instances in the last month for evaluation. FSIW: Feedback shift importance weighting [25]. Note that this method is not originally designed for streaming settings.…”

Section: Datasetsmentioning

confidence: 99%

Follow the Prophet: Accurate Online Conversion Rate Prediction in the Face of Delayed Feedback

Pan

et al. 2021

Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

The delayed feedback problem is one of the imperative challenges in online advertising, which is caused by the highly diversified feedback delay of a conversion varying from a few minutes to several days. It is hard to design an appropriate online learning system under these non-identical delay for different types of ads and users. In this paper, we propose to tackle the delayed feedback problem in online advertising by "Following the Prophet" (FTP for short).The key insight is that, if the feedback came instantly for all the logged samples, we could get a model without delayed feedback, namely the "prophet". Although the prophet cannot be obtained during online learning, we show that we could predict the prophet's predictions by an aggregation policy on top of a set of multi-task predictions, where each task captures the feedback patterns of different periods. We propose the objective and optimization approach for the policy, and use the logged data to imitate the prophet. Extensive experiments on three real-world advertising datasets show that our method outperforms the previous state-of-the-art baselines. CCS CONCEPTS• Computing methodologies → Machine learning.

show abstract

“…To learn the model from the bias distribution, Ktena et al [12] proposes two loss function FNW and FNC utilizing importance sampling Bottou et al [1], Sugiyama et al [21] to handle the distribution shift. Instead of labeling all samples as negative initially, Yasui et al [25] propose a feedback shift importance weighting algorithm, in which the model waits for the real conversion in a certain time interval. However, it does not allow the data correction, i.e., duplicated positive samples even a conversion event took place in the future.…”

Section: Delayed Feedback Modelsmentioning

confidence: 99%

Real Negatives Matter: Continuous Training with Real Negatives for Delayed Feedback Modeling

Gu¹,

Sheng²,

Fan³

et al. 2021

Preprint

View full text Add to dashboard Cite

One of the difficulties of conversion rate (CVR) prediction is that the conversions can delay and take place long after the clicks. The delayed feedback poses a challenge: fresh data are beneficial to continuous training but may not have complete label information at the time they are ingested into the training pipeline. To balance model freshness and label certainty, previous methods set a short waiting window or even do not wait for the conversion signal. If conversion happens outside the waiting window, this sample will be duplicated and ingested into the training pipeline with a positive label. However, these methods have some issues. First, they assume the observed feature distribution remains the same as the actual distribution. But this assumption does not hold due to the ingestion of duplicated samples. Second, the certainty of the conversion action only comes from the positives. But the positives are scarce as conversions are sparse in commercial systems. These issues induce bias during the modeling of delayed feedback. In this paper, we propose DElayed FEedback modeling with Real negatives (Defer) method to address these issues. The proposed method ingests real negative samples into the training pipeline. The ingestion of real negatives ensures the observed feature distribution is equivalent to the actual distribution, thus reducing the bias. The ingestion of real negatives also brings more certainty information of the conversion. To correct the distribution shift, Defer employs importance sampling to weigh the loss function. Experimental results on industrial datasets validate the superiority of Defer. Defer have been deployed in the display advertising system of Alibaba, obtaining over 6.0% improvement on CVR in several scenarios. The code and data in this paper are now open-sourced 1 .

show abstract

A Feedback Shift Correction in Predicting Conversion Rates under Delayed Feedback

Cited by 26 publications

References 15 publications

Asymptotically Unbiased Estimation for Delayed Feedback Modeling via Label Correction

Asymptotically Unbiased Estimation for Delayed Feedback Modeling via Label Correction

Follow the Prophet: Accurate Online Conversion Rate Prediction in the Face of Delayed Feedback

Real Negatives Matter: Continuous Training with Real Negatives for Delayed Feedback Modeling

Contact Info

Product

Resources

About