Objecting to consensual experiments even while approving of nonconsensual imposition of the policies they contain

Abstract

In previous survey studies using decision-making vignettes in which respondents rated and ranked the appropriateness of universally implementing one intervention for all relevant people (A), universally implementing another intervention (B), or conducting an experiment (or A/B test) to compare these interventions before implementing the superior one, we have found that people are often averse to the A/B test, even when they approve of implementing either of the interventions the experiment contains (Meyer et al., 2019, Heck et al., 2020). However, these vignettes did not say explicitly whether or not consent to the A/B test—or to universal imposition of either A or B, untested—would be obtained. In a preregistered analysis, when we asked respondents in four of our previous survey experiments to explain their ratings, 18% of those who rated the A/B test commented negatively on the apparent lack of consent, compared to fewer than 1% of participants who rated the A or B conditions offering the same objection (Meyer et al., 2019). Because implementing A and implementing B, collectively, entail imposing the same risks on people as conducting an A/B test, this contrast is puzzling. Moreover, pragmatic trials of the sort our vignettes describe are often not feasible to conduct with consent; as a result, IRBs often waive the default requirement of consent and ethicists have also argued that consent to research is not always required. Here, in three studies using previous materials, we test the effect on experiment aversion (Heck et al., 2020; cf. Mislavsky et al., 2019) and the A/B effect (Meyer et al., 2019) of stipulating that consent to the A/B test either will or will not be obtained. In both vignettes we test, we find that negative sentiments towards experiments persist even when it is explicitly stated that consent will be obtained. While these negative sentiments are substantially reduced when consent is present (compared to when consent is absent or not mentioned), significant experiment aversion and A/B effects remain: 29–31% of participants were averse to the experiment and 32–39% ranked it as the worst decision the agent could make—even when consent would be obtained from everyone involved in the A/B test and from no one involved in either policy imposition.

Experiment aversion does generalize, but it can also be mitigated

“…In 8 of 9 vignettes, more than 25% of participants ranked the A/B test as the worst option. Moreover, and contrary to MEM’s claim—in both their original paper and their reply—that people often prefer experiments, in none of their vignettes (or ours) was there significant experiment appreciation [when correctly defined as the inverse of EA, i.e., preferring the A/B test to the highest ( 8 , 9 )—not the lowest ( 1 )—rated policy]. Indeed, in several vignettes, significantly more participants were experiment-averse than experiment-appreciative.…”

“…Using this definition, a person shows experiment “preference” if they rate the A/B test higher than they rate their least-preferred policy. We believe that such a “preference” is not meaningful and instead calculate experiment appreciation which is defined as the difference between the rating of the A/B test and the rating of the highest-rated policy ( 8 , 9 ). Using this definition, a person shows experiment appreciation when they like the A/B test more than their favorite policy, a characteristic that we find meaningful.…”

“…For example, we previously identified lack of consent as a partial explanation for EA ( 3 ) and experimentally demonstrated that people are less averse to consensual experiments ( 8 ). In all MEM studies [except their successful direct replication of Meyer et al ( 3 )], participants are told “[t]here are 3 hospitals you can choose to be treated at” and the first and primary dependent measure asks, “How likely are you to choose to be treated at this hospital?” That people are less averse to consensual pragmatic trials is unsurprising, not especially actionable [since consent reduces external validity, is typically impractical, and hence is absent from most corporate and many pragmatic healthcare trials ( 10 )], and not evidence that EA fails to generalize.…”

“…In our work and others’ ( 13 , 14 ), EA generalizes across domains [medical, public health, public policy, technology; ( 3 , 7 – 9 , 13 )], scenarios [safety checklists for catheterization and intubation, prescribing hypertensive and corticosteroid drugs, return of results from genetic testing, retirement savings plans, overrides for autonomous vehicles, ventilator proning for COVID patients, post-COVID school reopening, rules for wearing masks during COVID, distribution of COVID vaccines, recruitment of health workers, poverty alleviation strategies, teacher well-being strategies, basic income plans, lead abatement strategies; ( 3 , 7 – 9 , 13 )], populations [laypeople, clinicians, public sector leaders; ( 3 , 9 , 13 )], and levels of consent [conducting the A/B test after obtaining consent, conducting it without asking for consent, and silence about whether consent was sought; ( 8 )]. MEM themselves show that EA generalizes across seven dependent measures [(in)appropriate, (un)ethical, (ir)responsible, (un)professional, (un)informed, backfire/succeed, likely to choose].…”

Reply to Vogt et al.: What does it mean that a phenomenon generalizes?

“… 5 pre-registered the aim to answer whether EA continues to emerge when respondents have complete information about all treatment options and found that it does. Also, while a new preprint ( 11 ) manipulated consent, it found that decisions for consensual experiments continued to be evaluated as relatively less appropriate. Importantly, given VHWCM’s classification, all our between-subject designs were in fact nonconsensual ( Table 1 ), yet they did not reveal relatively lower appropriateness-ratings.…”