A Large Scale Benchmark for Individual Treatment Effect Prediction and Uplift Modeling

Abstract

Individual Treatment Effect (ITE) prediction is an important area of research in machine learning which aims at explaining and estimating the causal impact of an action at the granular level. It represents a problem of growing interest in multiple sectors of application such as healthcare, online advertising or socioeconomics. To foster research on this topic we release a publicly available collection of 13.9 million samples collected from several randomized control trials, scaling up previously available datasets by a healthy 210x factor. We provide details on the data collection and perform sanity checks to validate the use of this data for causal inference tasks. First, we formalize the task of uplift modeling (UM) that can be performed with this data, along with the relevant evaluation metrics. Then, we propose synthetic response surfaces and heterogeneous treatment assignment providing a general set-up for ITE prediction. Finally, we report experiments to validate key characteristics of the dataset leveraging its size to evaluate and compare − with high statistical significance − a selection of baseline UM and ITE prediction methods.

Maximizing the Success Probability of Policy Allocations in Online Systems

“…• CRITEO-UPLIFT v2 (Diemert et al 2021) is provided by the AdTech company Criteo. Data contains 13.9 million samples which are collected from several incremental A/B tests.…”

Shenzhen, China

“…These include three aspects that affect the difficulty of use: benchmark results, hyperparameter search, and optimal parameters provided; two aspects of data preprocessing settings, instance deduplication and feature normalization; and one aspect that measures the degree of compatibility with DUM. As shown in Table 1, we evaluate a set of existing benchmarks and libraries for uplift modeling and causal inference that are powerfully relevant to our work against these considerations, including Criteo-ITE-Benchmark [13], CATENets [10], DoWhy [27], EconML [5], CausalML [7], DECI 1 , ShowWhy 2 . We find that none of them fully meet these requirements, which motivates us to propose a new public benchmark for DUM in this paper.…”

“…It provides data sets from two different sources, where the training set is collected from a production environment with a treatment bias, the treatment allocation is selective due to the operation target policy, and the test set is collected from the users who are not affected by the targeting strategy and the treatment assignment follows the randomized controlled trials (RCT). 2) Criteo [13] is a dataset from real advertising scenarios provided by Criteo AI Labs. It contains nearly 14 million instances with similar treatment bias, 12 continuous features, a treatment indicator, and 2 labels (i.e., visits and conversions).…”

“…Despite the promising results from these aforementioned studies, there is a lack of standardized benchmarks and unified evaluation protocols for DUM tasks. For example, although industrial benchmark datasets such as Criteo [13] and Lazada [34] are used in existing studies, these works often use different random seeds to perform unknown train-test splits. In addition, there is no consistent standard for data preprocessing steps, such as feature normalization, instance deduplication, etc, and discussion of their potential impacts is neglected.…”

“…Our work standardizes the open benchmarking pipeline for such modeling and thoroughly compares different DUM models for reproducible studies. We extensively evaluate 13 existing representative models in multiple data processing settings on two widely-used representative industrial datasets, including Criteo [13] and Lazada [34]. The former contains training and test subsets with instance selection bias, and the latter contains a training subset with instance selection bias and a test subset without bias.…”