A Counterfactual Framework for Seller-Side A/B Testing on Marketplaces

Ha-Thuc, Viet; Dutta, Avishek; Mao, Ren; Wood, Matthew; Liu, Yunli

doi:10.1145/3397271.3401434

Cited by 14 publications

(13 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent work has developed methods to minimize this bias using modified randomization schemes [4,13], experiment designs where treatment is incrementally applied to a market (e.g., small pricing changes) [23], and designs that randomize on both sides of the market [2,16]. Specialized designs have also been designed for particular interventions, such modifications in ranking algorithms [12].…”

Section: Related Workmentioning

confidence: 99%

Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms

Li¹,

Geng²,

Johari³

et al. 2021

Preprint

View full text Add to dashboard Cite

Two-sided marketplace platforms often run experiments (or A/B tests) to test the effect of an intervention before launching it platform-wide. A typical approach is to randomize individuals into the treatment group, which receives the intervention, and the control group, which does not. The platform then compares the performance in the two groups to estimate the effect if the intervention were launched to everyone. We focus on two common experiment types, where the platform randomizes individuals either on the supply side or on the demand side. For these experiments, it is well known that the resulting estimates of the treatment effect are typically biased: because individuals in the market compete with each other, individuals in the treatment group affect individuals in the control group (and vice versa), creating interference and leading to a biased estimate.We develop a simple, tractable market model to study bias and variance in these experiments with interference. We focus on two choices available to the platform: (1) Which side of the platform should it randomize on (supply or demand)? (2) What proportion of individuals should be allocated to treatment? We find that both choices affect the bias and variance of the resulting estimators, but in different ways. The bias-optimal choice of experiment type depends on the relative amounts of supply and demand in the market, and we discuss how a platform can use market data to select the experiment type. Importantly, we find that in many circumstances, choosing the bias-optimal experiment type has little effect on variance, and in some cases can coincide with the variance-optimal type. On the other hand, we find that the choice of treatment proportion can induce a bias-variance tradeoff, where the bias-minimizing proportion actually increases variance. We discuss how a platform can navigate this tradeoff and best choose the treatment proportion, using a combination of modeling as well as contextual knowledge about the market, the risk of the intervention, and reasonable effect sizes of the intervention.

show abstract

Section: Related Workmentioning

confidence: 99%

Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms

Li¹,

Geng²,

Johari³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Let us consider a bipartite graph linking two types of entities -producers and consumers. A recommender system recommends an ordered set of items generated by the producers to each consumer, where items (e.g., connection recommendations, content recommendations or search recommendations) are ordered based on their estimated relevance in that consumer session 3 . We use the terminology "consumer (or producer) side experience" to refer to a measurable quantity associated with a consumer (or producer) that depends on the rank assigned by the recommendation system.…”

Section: Problem Setupmentioning

confidence: 99%

“…First, we compare the design accuracy and the cost of the variants of U niCoRn(α) based on a number of values of α. Next, we compare the performances of U niCoRn(α) for α ∈ {0, 0.2, 1}, the counterfactual ranking method of [3] (we will refer to this as HaT hucEtAl) and a modified version of OASIS [9] for estimating the average treatment effect. To the best of our knowledge, these are the only existing methods that do not require the underlying network to be known a priori.…”

Section: Empirical Evaluationmentioning

confidence: 99%

“…By unifying the different counterfactual rankings (hence "UniCoRn") based on treatment allocation on the producer side, and careful handling of conflicts, we obtain a high quality experiment. A recent work [3] is a specific instance of our generalized design, relying on small producer ramps, which minimize chances of conflict between the counterfactual rankings.…”

Section: Introductionmentioning

confidence: 99%

“…• We show through extensive simulations how the method performs in various synthetic scenarios and against multiple existing approaches [3,9].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A/B Testing for Recommender Systems in a Two-sided Marketplace

Nandy¹,

Venugopalan²,

Lo³

et al. 2021

Preprint

View full text Add to dashboard Cite

Two-sided marketplaces are standard business models of many online platforms (e.g., Amazon, Facebook, LinkedIn), wherein the platforms have consumers, buyers or content viewers on one side and producers, sellers or content-creators on the other. Consumer side measurement of the impact of a treatment variant can be done via simple online A/B testing. Producer side measurement is more challenging because the producer experience depends on the treatment assignment of the consumers. Existing approaches for producer side measurement are either based on graph cluster-based randomization or on certain treatment propagation assumptions. The former approach results in low-powered experiments as the producer-consumer network density increases and the latter approach lacks a strict notion of error control. In this paper, we propose (i) a quantification of the quality of a producer side experiment, and (ii) a new experiment design mechanism that generates high quality experiments based on this quantification. Our approach, called UniCoRn (Unifying Counterfactual Rankings), provides explicit control over the quality of the experiment and its computation cost. Further, we prove that our experiment design is optimal. Our approach is agnostic to the density of the producer-consumer network and does not rely on any treatment propagation assumption. Moreover, unlike the existing approaches, we do not need to know the underlying network in advance, making this widely applicable to the industrial setting where the underlying network is unknown and challenging to predict a priori due to its dynamic nature. We use simulations to thoroughly validate our approach and compare it against existing methods. We also implement UniCoRn in an edge recommendation application that serves tens of millions of members and billions of edge recommendations daily.A/B testing is a powerful tool because of its design simplicity and ease of setup. It is accurate in applications where the behavior of a measurement unit (e.g., a user) is unaffected by the treatment allocated to any other measurement unit. This principle is called "Stable Unit Treatment Value Assumption" or SUTVA [11][12][13]. The SUTVA principle is reasonably accurate for experiments in several viewer side applications of recommender systems (e.g., newsfeed ranking, search), where each viewer acts independently based only on what is shown to her.Preprint. Under review.

show abstract