Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms

Li, Hannah; Geng, Zhao; Johari, Ramesh; Weintraub, Gabriel Y.

doi:10.1145/3485447.3512063

Cited by 13 publications

(10 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This design first clusters units according to the provided graph, and then assigns each cluster to either treatment or control. In the specific case of bipartite graphs, other works propose modifications to unit-level randomization by choosing which side of the graph to randomize [22,24,8]. The authors of [29,19] look directly at clustered designs on bipartite graphs, but they study a two-sided experimental framework in which one side of the graph is randomized while the other is measured.…”

Section: Related Workmentioning

confidence: 99%

“…We test the robustness of our experimental design to misspecification of the exposure mapping by simulating outcomes according to a model developed for vacation rentals by Li et al [24], in which there is no explicit exposure mapping defined. We call the experimental units customers and the interference units listings.…”

Section: Robustness To Exposure Mapping: An Airbnb Case Studymentioning

confidence: 99%

“…The measured outcome Y i is 1 if customer i successfully booked a listing, and 0 otherwise. In alignment with previous work in this literature [24,22] we create a natural clustering structure on the network by assigning each customer and each listing to one of 20 types. Our simulation takes N = 500 customers and M = 1000 listings, with application probability under the control assignment of φ is = 0.016 if i and s are of the same type, and φ is = 0.0001 otherwise.…”

Section: Robustness To Exposure Mapping: An Airbnb Case Studymentioning

confidence: 99%

“…All clustering designs used K = 20 clusters, with K T = 10 assigned to treatment. Simulations were performed by extending code provided by the authors of Li et al [24] to the cluster-randomized setting. The relative bias, standard deviation, and root mean squared error (RMSE) of τ DIM of various clustering algorithms are shown in Figure 2.…”

Section: Robustness To Exposure Mapping: An Airbnb Case Studymentioning

confidence: 99%

“…Formally a violation of the Stable Unit Treatment Value Assumption [31], interference has been studied in many settings, including agricultural studies [27], clinical trials [20], social networks [chap. 16 of 5, 2, 17, 16], and marketplaces [10,30,24,37,26]. Marketplaces exhibit unique forms of interference because they involve two types of units: buyers and sellers.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Cluster Randomized Designs for One-Sided Bipartite Experiments

Brennan¹,

Mirrokni²,

Pouget-Abadie³

2022

Preprint

View full text Add to dashboard Cite

The conclusions of randomized controlled trials may be biased when the outcome of one unit depends on the treatment status of other units, a problem known as interference. In this work, we study interference in the setting of one-sided bipartite experiments in which the experimental units-where treatments are randomized and outcomes are measured-do not interact directly. Instead, their interactions are mediated through their connections to interference units on the other side of the graph. Examples of this type of interference are common in marketplaces and two-sided platforms. The cluster-randomized design is a popular method to mitigate interference when the graph is known, but it has not been well-studied in the one-sided bipartite experiment setting. In this work, we formalize a natural model for interference in one-sided bipartite experiments using the exposure mapping framework. We first exhibit settings under which existing cluster-randomized designs fail to properly mitigate interference under this model. We then show that minimizing the bias of the difference-in-means estimator under our model results in a balanced partitioning clustering objective with a natural interpretation. We further prove that our design is minimax optimal over the class of linear potential outcomes models with bounded interference. We conclude by providing theoretical and experimental evidence of the robustness of our design to a variety of interference graphs and potential outcomes models.

show abstract