Experiment aversion among clinicians and the public — an obstacle to evidence-based medicine and public health

Abstract

Background: Randomized controlled trials (RCTs) are essential for determining the safety and efficacy of healthcare interventions. However, both laypeople and clinicians often demonstrate experiment aversion: preferring to implement either of two interventions for everyone rather than comparing them to determine which is best. We studied whether clinician and layperson views of pragmatic RCTs for Covid-19 or other interventions became more positive early in the pandemic, which increased both the urgency and public discussion of RCTs. Methods: We conducted several survey studies with laypeople (total n=2,909) and two with clinicians (n=895; n=1,254) in 2020 and 2021. Participants read vignettes in which a hypothetical decision-maker who sought to improve health could choose to implement intervention A for all, implement intervention B for all, or experimentally compare A and B and implement the superior intervention. Participants rated and ranked the appropriateness of each decision. Results: Compared to our pre-pandemic results, we found no decrease in laypeople's aversion to non-Covid-19 experiments involving catheterization checklists and hypertension drugs. Nor were either laypeople or clinicians less averse to Covid-19 RCTs (concerning corticosteroid drugs, vaccines, intubation checklists, proning, school reopening, and mask protocols), on average. Across all vignettes and samples, levels of experiment aversion ranged from 28% to 57%, while levels of experiment appreciation (in which the RCT is rated higher than the participant's highest-rated intervention) ranged from only 6% to 35%. Conclusions: Advancing evidence-based medicine through pragmatic RCTs will require anticipating and addressing experiment aversion among both patients and healthcare professionals.

Experiment aversion does generalize, but it can also be mitigated

“…In 8 of 9 vignettes, more than 25% of participants ranked the A/B test as the worst option. Moreover, and contrary to MEM’s claim—in both their original paper and their reply—that people often prefer experiments, in none of their vignettes (or ours) was there significant experiment appreciation [when correctly defined as the inverse of EA, i.e., preferring the A/B test to the highest ( 8 , 9 )—not the lowest ( 1 )—rated policy]. Indeed, in several vignettes, significantly more participants were experiment-averse than experiment-appreciative.…”

“…In our work and others’ ( 13 , 14 ), EA generalizes across domains [medical, public health, public policy, technology; ( 3 , 7 – 9 , 13 )], scenarios [safety checklists for catheterization and intubation, prescribing hypertensive and corticosteroid drugs, return of results from genetic testing, retirement savings plans, overrides for autonomous vehicles, ventilator proning for COVID patients, post-COVID school reopening, rules for wearing masks during COVID, distribution of COVID vaccines, recruitment of health workers, poverty alleviation strategies, teacher well-being strategies, basic income plans, lead abatement strategies; ( 3 , 7 – 9 , 13 )], populations [laypeople, clinicians, public sector leaders; ( 3 , 9 , 13 )], and levels of consent [conducting the A/B test after obtaining consent, conducting it without asking for consent, and silence about whether consent was sought; ( 8 )]. MEM themselves show that EA generalizes across seven dependent measures [(in)appropriate, (un)ethical, (ir)responsible, (un)professional, (un)informed, backfire/succeed, likely to choose].…”

“…Using this definition, a person shows experiment “preference” if they rate the A/B test higher than they rate their least-preferred policy. We believe that such a “preference” is not meaningful and instead calculate experiment appreciation which is defined as the difference between the rating of the A/B test and the rating of the highest-rated policy ( 8 , 9 ). Using this definition, a person shows experiment appreciation when they like the A/B test more than their favorite policy, a characteristic that we find meaningful.…”