Design and Analysis of Switchback Experiments

Bojinov, Iavor; Simchi‐Levi, David; Zhao, Jinglong

doi:10.2139/ssrn.3684168

Cited by 15 publications

(12 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Simply comparing the work practices and/ or productivity levels of remote workers and office workers will likely yield biased estimates of the global treatment effects of different remote work policies, due to the causal effects of one's colleagues working remotely. In conducting these experiments, it is crucial that firms use experiment designs that are optimized for capturing the overall effects of remote work policies, for example, graph cluster randomization 53,54 or switchback randomization 55 . Ideally, such field experiments would be complemented with high-quality qualitative research that can describe emergent processes and workers' perceptions and, more generally, uncover insights beyond those that can be obtained through quantitative methods.…”

Section: Discussionmentioning

confidence: 99%

The effects of remote work on collaboration among information workers

et al. 2021

View full text Add to dashboard Cite

Section: Discussionmentioning

confidence: 99%

The effects of remote work on collaboration among information workers

et al. 2021

View full text Add to dashboard Cite

“…One prominent design with a long history is the crossover design or switchback design, see Cochran [1939], Cox [1948], Brown Jr [1980], Cook and DeMets [2007], Bojinov et al [2020]. In such designs, each unit is assigned to both treatment and control, but at different times.…”

Section: Related Literaturementioning

confidence: 99%

Multiple Randomization Designs

Bajari¹,

Burdick²,

Imbens³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this study we introduce a new class of experimental designs. In a classical randomized controlled trial (RCT), or A/B test, a randomly selected subset of a population of units (e.g., individuals, plots of land, or experiences) is assigned to a treatment (treatment A), and the remainder of the population is assigned to the control treatment (treatment B). The difference in average outcome by treatment group is an estimate of the average effect of the treatment. However, motivating our study, the setting for modern experiments is often different, with the outcomes and treatment assignments indexed by multiple populations. For example, outcomes may be indexed by buyers and sellers, by content creators and subscribers, by drivers and riders, or by travelers and airlines and travel agents, with treatments potentially varying across these indices. Spillovers or interference can arise from interactions between units across populations. For example, sellers' behavior may depend on buyers' treatment assignment, or vice versa. This can invalidate the simple comparison of means as an estimator for the average effect of the treatment in classical RCTs. We propose new experiment designs for settings in which multiple populations interact. We show how these designs allow us to study questions about interference that cannot be answered by classical randomized experiments. Finally, we develop new statistical methods for analyzing these Multiple Randomization Designs. * We are grateful for discussions with Susan Athey, John Geweke, Matt Taddy, and Johan Ugander, and for comments by participants in the CODE@MIT, NABE (2019), and the ASSA conferences (2020). This research was carried out as part of Imbens' consulting relationship with Amazon. Brian Burdick contributed while working for Amazon.

show abstract

“…In practice, platforms looking to minimize interference bias generally run clustered randomized designs [8], in which the unit of observation is changed, or switchback testing [21], where the treatment is turned on and off over time. Both approaches create a large increase in variance due to the reduction in sample size, and recent work has aimed to minimize this variance in switchback designs [11,6]. Still, due to variance concerns and ease of implementation, simpler CR and LR designs are often used in practice despite the bias that can arise.…”

Section: Related Workmentioning

confidence: 99%

“…where ψ and ψ take the exact same form as defined in (6). Combining (20) and ( 21) gives the limit (15).…”

Section: A Proofsmentioning

confidence: 99%

“…The existence of interference on marketplace experiments is well documented (see related work below); indeed, previous studies have shown that the resulting bias can be as large as the GTE itself [5,14,10]. To address this problem, many platforms have adopted alternative experiment designs such as clustered experiments or switchback experiments, where the platform randomizes on geographical units or intervals of time instead of randomizing individuals [22,8,21,6]. These types of experiments can decrease bias but also increase variance.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms

Li¹,

Geng²,

Johari³

et al. 2021

Preprint

View full text Add to dashboard Cite

Two-sided marketplace platforms often run experiments (or A/B tests) to test the effect of an intervention before launching it platform-wide. A typical approach is to randomize individuals into the treatment group, which receives the intervention, and the control group, which does not. The platform then compares the performance in the two groups to estimate the effect if the intervention were launched to everyone. We focus on two common experiment types, where the platform randomizes individuals either on the supply side or on the demand side. For these experiments, it is well known that the resulting estimates of the treatment effect are typically biased: because individuals in the market compete with each other, individuals in the treatment group affect individuals in the control group (and vice versa), creating interference and leading to a biased estimate.We develop a simple, tractable market model to study bias and variance in these experiments with interference. We focus on two choices available to the platform: (1) Which side of the platform should it randomize on (supply or demand)? (2) What proportion of individuals should be allocated to treatment? We find that both choices affect the bias and variance of the resulting estimators, but in different ways. The bias-optimal choice of experiment type depends on the relative amounts of supply and demand in the market, and we discuss how a platform can use market data to select the experiment type. Importantly, we find that in many circumstances, choosing the bias-optimal experiment type has little effect on variance, and in some cases can coincide with the variance-optimal type. On the other hand, we find that the choice of treatment proportion can induce a bias-variance tradeoff, where the bias-minimizing proportion actually increases variance. We discuss how a platform can navigate this tradeoff and best choose the treatment proportion, using a combination of modeling as well as contextual knowledge about the market, the risk of the intervention, and reasonable effect sizes of the intervention.

show abstract

Design and Analysis of Switchback Experiments

Cited by 15 publications

References 38 publications

The effects of remote work on collaboration among information workers

The effects of remote work on collaboration among information workers

Multiple Randomization Designs

Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms

Contact Info

Product

Resources

About