Understanding and Misunderstanding Randomized Controlled Trials

Abstract

RCTs would be more useful if there were more realistic expectations of them and if their pitfalls were better recognized. For example, and contrary to many claims in the applied literature, randomization does not equalize everything but the treatment across treatments and controls, it does not automatically deliver a precise estimate of the average treatment effect (ATE), and it does not relieve us of the need to think about (observed or unobserved) confounders. Estimates apply to the trial sample only, sometimes a convenience sample, and usually selected; justification is required to extend them to other groups, including any population to which the trial sample belongs. Demanding "external validity" is unhelpful because it expects too much of an RCT while undervaluing its contribution. Statistical inference on ATEs involves hazards that are not always recognized. RCTs do indeed require minimal assumptions and can operate with little prior knowledge. This is an advantage when persuading distrustful audiences, but it is a disadvantage for cumulative scientific progress, where prior knowledge should be built upon and not discarded. RCTs can play a role in building scientific knowledge and useful predictions but they can only do so as part of a cumulative program, combining with other methods, including conceptual and theoretical development, to discover not "what works," but "why things work".

Social determinants of health and the International Monetary Fund

“…household's socioeconomic status. Because emerging evidence suggests that it may be the latter, understanding and testing the specific pathways (1) through which education impacts health might be more useful and productive (14). Disentangling and intervening on the context-specific pathways of education is likely to provide an opportunity to impact child health, particularly because uneducated adult parents are unlikely to return to school and gain a specific level of formal educational attainment.…”

Social Norms and Energy Conservation Beyond the US

“…On a more general note, our paper contributes to the emerging literature on the transferability of causal effects across settings (Allcott 2015, Deaton and Cartwright 2016, Gechter 2016, Hotz et al 2005, Leviton 2017, Muller 2015, Peters et al 2016, Pritchett et al 2015, Vivalt 2015.…”

Building Capacity for Data-Driven Governance: Creating a New Foundation for Democracy

“…Their uses range from descriptive displays such as dashboards, (e.g., Robert Wood Johnson Foundation 2017), to randomized control trials, (e.g., Abdul Latif Jameel Poverty Action Lab 2017), to draw inferences. Randomized control studies can provide statistical evidence as to whether or not a program works in a specific context, but are unable to explain why (Behn 2015, Deaton et al 2016. Randomized control studies are valuable but insufficient to explain specific social phenomena.…”