Offline A/B Testing for Recommender Systems

Abstract

Online A/B testing evaluates the impact of a new technology by running it in a real production environment and testing its performance on a subset of the users of the platform. It is a well-known practice to run a preliminary offline evaluation on historical data to iterate faster on new ideas, and to detect poor policies in order to avoid losing money or breaking the system. For such offline evaluations, we are interested in methods that can compute offline an estimate of the potential uplift of performance generated by a new technology. Offline performance can be measured using estimators known as counterfactual or off-policy estimators. Traditional counterfactual estimators, such as capped importance sampling or normalised importance sampling, exhibit unsatisfying bias-variance compromises when experimenting on personalized product recommendation systems. To overcome this issue, we model the bias incurred by these estimators rather than bound it in the worst case, which leads us to propose a new counterfactual estimator. We provide a benchmark of the different estimators showing their correlation with business metrics observed by running online A/B tests on a large-scale commercial recommender system. CCS CONCEPTS• Computing methodologies → Learning from implicit feedback; • Information systems → Evaluation of retrieval results;

Off-Policy Evaluation of Probabilistic Identity Data in Lookalike Modeling

“…We thus Here we use weight truncation [31], w T (w) = min(w, w 0 ), where weights are truncated as they cross a threshold w 0 . While many other works [15,20] have focused on comparing a wide variety of counterfactual estimators (e.g., weight dropping, truncation, model-based and doubly-robust variants) we opted not to do this here, as our main focus is on the evaluation of the impact of identity data, and as we were not able to follow up with the kind of extensive A/B testing that would be required to definitively judge the relative performance of various estimators. We did do our analysis twice, once using the weight truncation mentioned above, and once using weight dropping [5], where weights are set to zero upon crossing a threshold w 0 , finding estimates that were consistent with w T and of similar variance (these are omitted for clarity in what follows).…”

Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems

“…To perform evaluation of RL methods on ground-truth, a straightforward way is to evaluate the learned policy through online A/B test, which, however, could be prohibitively expensive and may hurt user experiences. Suggested by [6,7], a specific off-policy estimator NCIS [25] is employed to evaluate the performance of different recommender agents. The step-wise variant of NCIS is defined aŝ ρ t 1 :t 2 = min{c,…”

Estimating Error and Bias in Offline Evaluation Results

“…Counterfactual evaluation [5,11,20] uses causal inference techniques -often inverse propensity scoring -to estimate how users would have responded to a different recommender algorithm. However, it is difficult to apply to commonly-used data sets and does not yield insight into the reliability of existing evaluations.…”