Design and Analysis of Cluster-Randomized Field Experiments in Panel Data Settings

Abstract

Field experiments conducted with the village, city, state, region, or even country as the unit of randomization are becoming commonplace in the social sciences. While convenient, subsequent data analysis may be complicated by the constraint on the number of clusters in treatment and control. Through a battery of Monte Carlo simulations, we examine best practices for estimating unit-level treatment effects in cluster-randomized field experiments, particularly in settings that generate short panel data. In most settings we consider, unit-level estimation with unit fixed effects and cluster-level estimation weighted by the number of units per cluster tend to be robust to potentially problematic features in the data while giving greater statistical power. Using insights from our analysis, we evaluate the effect of a unique field experiment: a nationwide tipping field experiment across markets on the Uber app. Beyond the import of showing how tipping affects aggregate market outcomes, we provide several insights on aspects of generating and analyzing cluster-randomized experimental data when there are constraints on the number of experimental units in treatment and control.

The Dozen Things Experimental Economists Should Do (More of)

“…However, adding a participant from a new cluster tends to be more expensive than another participant from an existing cluster. Additionally, Chandar et al (2018) argue that in case of heterogeneous treatment effects at the cluster level, the researcher may want to include more treated clusters than control clusters. 56 We return to the question of treatment assignment in 56 The intuition is that if the researcher's intervention leads to different effects across different clusters, having more treated clusters can help average over those differences and recover the mean effect (Chandar et al 2018). sections 3.8 and 3.9 where we discuss in detail the practice of blocked randomization and within subject (WS) designs.…”

“…However, adding a participant from a new cluster tends to be more expensive than another participant from an existing cluster. Additionally, Chandar et al () argue that in case of heterogeneous treatment effects at the cluster level, the researcher may want to include more treated clusters than control clusters . We return to the question of treatment assignment in sections 3.8 and 3.9 where we discuss in detail the practice of blocked randomization and within subject (WS) designs.…”

“…The intuition is that if the researcher's intervention leads to different effects across different clusters, having more treated clusters can help average over those differences and recover the mean effect (Chandar et al ).…”

The Drivers of Social Preferences: Evidence from a Nationwide Tipping Field Experiment

“…The rollout was staggered across cities in part to ensure there were no bugs in the product and in part as an experiment at the city level to understand tipping's impact on the marketplace (see Chandar et al (2019) for results from this market-level field experiment). We randomized three cities to receive tipping on June 20, 2017 (internally called the alpha launch), followed by half of the operational markets in the United States and Canada on July 6, 2017 (internally called the beta launch).…”

Workplace Knowledge Flows

“…Here the managerweek level comes from fixed manager groupings at the time of assignment, and agents in the original group are averaged together even if working under a different manager in the post-intervention period. Columns 5-8 weight by the number of agents on a team in each week, as suggested by Chandar et al (2019). Standard errors are clustered at the sales manager level and are reported in parentheses.…”