Reply to Mislavsky et al.: Sometimes people really are averse to experiments

Meyer, Michelle N.; Heck, Patrick R.; Holtzman, Geoffrey S.; Anderson, Sharolyn; Cai, William; Watts, Duncan J.; Chabris, Christopher F.

doi:10.1073/pnas.1914509116

Cited by 10 publications

(21 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In School Reopening, experiment aversion was not observed because participants on average clearly preferred intervention B to A and rated the experiment similar to intervention A. 26,27 53% of participants ranked intervention B as the best of the three options (compared to 17% choosing intervention A and 30% choosing the A/B test).…”

Section: Lay Sentiments About Covid-19 Specific Healthcare Experiment...mentioning

confidence: 99%

Experiment aversion among clinicians and the public — an obstacle to evidence-based medicine and public health

Vogt

Heck

Mestechkin

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Background: Randomized controlled trials (RCTs) are essential for determining the safety and efficacy of healthcare interventions. However, both laypeople and clinicians often demonstrate experiment aversion: preferring to implement either of two interventions for everyone rather than comparing them to determine which is best. We studied whether clinician and layperson views of pragmatic RCTs for Covid-19 or other interventions became more positive early in the pandemic, which increased both the urgency and public discussion of RCTs. Methods: We conducted several survey studies with laypeople (total n=2,909) and two with clinicians (n=895; n=1,254) in 2020 and 2021. Participants read vignettes in which a hypothetical decision-maker who sought to improve health could choose to implement intervention A for all, implement intervention B for all, or experimentally compare A and B and implement the superior intervention. Participants rated and ranked the appropriateness of each decision. Results: Compared to our pre-pandemic results, we found no decrease in laypeople's aversion to non-Covid-19 experiments involving catheterization checklists and hypertension drugs. Nor were either laypeople or clinicians less averse to Covid-19 RCTs (concerning corticosteroid drugs, vaccines, intubation checklists, proning, school reopening, and mask protocols), on average. Across all vignettes and samples, levels of experiment aversion ranged from 28% to 57%, while levels of experiment appreciation (in which the RCT is rated higher than the participant's highest-rated intervention) ranged from only 6% to 35%. Conclusions: Advancing evidence-based medicine through pragmatic RCTs will require anticipating and addressing experiment aversion among both patients and healthcare professionals.

show abstract

Section: Lay Sentiments About Covid-19 Specific Healthcare Experiment...mentioning

confidence: 99%

Experiment aversion among clinicians and the public — an obstacle to evidence-based medicine and public health

Vogt

Heck

Mestechkin

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The evidence for relative experiment aversion published by Meyer et al. ( 5 , 6 ) and Heck et al. ( 7 ) appears “robust and general” and often “substantial” (ref.…”

mentioning

confidence: 97%

“…However, recent works by Meyer et al. ( 5 , 6 ) and Heck et al. ( 7 ) raise concerns about a significant barrier to evidence-based practice: relative Experiment Aversion .…”

mentioning

confidence: 99%

“…The conclusion put forward by the recent work ( 5 – 7 ) has potentially wide-reaching negative implications for evidence-based policy. As the authors argue, individuals’ attitudes toward experiments matter not because these individuals decide on policy implementation, but because they affect policymakers’ decisions; policymakers who anticipate objections toward experiments “may forgo them in favor of universal implementation or may conduct randomized evaluations in secret, neither of which is optimal” (ref.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Experiment aversion does not appear to generalize

Mažar

Elbæk

Mitkidis

2023

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Over the past decade, governments and organizations around the world have established behavioral insights teams advocating for randomized experiments. However, recent findings by M. N. Meyer et al. , Proc. Natl. Acad. Sci. U.S.A. 116 , 10723–10728 (2019) and P. R. Heck, C. F. Chabris, D. J. Watts, M. N. Meyer, Proc. Natl. Acad. Sci. U.S.A. 117 , 18948–18950 (2020) suggest that people often rate randomized experiments as less appropriate than the policies they contain even when approving the implementation of either policy untested and when none of the individual policies is clearly superior. The authors warn that this could cause policymakers to avoid running large-scale field experiments or being transparent about running them and might contribute to an adverse heterogeneity bias in terms of who is participating in experiments. In one direct and six conceptual preregistered replications (total N = 5,200) of the previously published larger-effect studies, using the same main dependent variable but with variations in scenario wordings, recruitment platforms, and countries, and the addition of further measures to assess people’s views, we test the generalizability and robustness of these findings. Together, we find that the original results do not appear to generalize. That is, our triangulation reveals insufficient evidence to conclude that people exhibit a common pattern of behavior that would be consistent with relative experiment aversion, thereby supporting recent findings by R. Mislavsky, B. Dietvorst, U. Simonsohn, Mark. Sci. 39 , 1092–1104 (2020). Thus, policymakers may not need to be concerned about employing evidence-based practices more so than about universally implementing policies.

show abstract

“…If that were the only reason why people object to experiments, then policy makers could theoretically forestall backlashes to A/B testing by only comparing policies that people like, as Dietvorst et al (10) suggest. But, if people sometimes object to A/B tests more than they object to either of the policies these tests compare, and absent any rational reasons that might exist for objecting to particular experiments, such a pattern may threaten evidence-based practices and policy by reflecting a genuine aversion to randomized evaluation (3,11).…”

mentioning

confidence: 99%

Objecting to experiments even while approving of the policies or treatments they compare

Heck

Chabris

Watts

et al. 2020

Proc. Natl. Acad. Sci. U.S.A.

Self Cite

View full text Add to dashboard Cite

We resolve a controversy over two competing hypotheses about why people object to randomized experiments: 1) People unsurprisingly object to experiments only when they object to a policy or treatment the experiment contains, or 2) people can paradoxically object to experiments even when they approve of implementing either condition for everyone. Using multiple measures of preference and test criteria in five preregistered within-subjects studies with 1,955 participants, we find that people often disapprove of experiments involving randomization despite approving of the policies or treatments to be tested.

show abstract

Reply to Mislavsky et al.: Sometimes people really are averse to experiments

Cited by 10 publications

References 3 publications

Experiment aversion among clinicians and the public — an obstacle to evidence-based medicine and public health

Experiment aversion among clinicians and the public — an obstacle to evidence-based medicine and public health

Experiment aversion does not appear to generalize

Objecting to experiments even while approving of the policies or treatments they compare

Contact Info

Product

Resources

About