The statistical pitfalls of the partially randomized preference design in non‐blinded trials of psychological interventions

Abstract

In a partially randomized preference trial (PRPT) patients with no treatment preference are allocated to groups at random, but those who express a preference receive the treatment of their choice. It has been suggested that the design can improve the external and internal validity of trials. We used computer simulation to illustrate the impact that an unmeasured confounder could have on the results and conclusions drawn from a PRPT. We generated 4000 observations ("patients") that reflected the distribution of the Beck Depression Index (DBI) in trials of depression. Half were randomly assigned to a randomized controlled trial (RCT) design and half were assigned to a PRPT design. In the RCT, "patients" were evenly split between treatment and control groups; whereas in the preference arm, to reflect patient choice, 87.5% of patients were allocated to the experimental treatment and 12.5% to the control. Unadjusted analyses of the PRPT data consistently overestimated the treatment effect and its standard error. This lead to Type I errors when the true treatment effect was small and Type II errors when the confounder effect was large. The PRPT design is not recommended as a method of establishing an unbiased estimate of treatment effect due to the potential influence of unmeasured confounders.

Pre-ordering lunch at work. Results of the what to eat for lunch study

“…We chose to include all consenting participants in our analyses, which included three participants added to the study post randomization. This is a limitation of our study because partial randomization can lead to biased estimates of treatment effects (Gemmell & Dunn, 2011). Sensitivity analyses suggested our contemporaneous comparisons may underestimate the differences between our study groups.…”

Patients’ treatment expectancies in clinical trials of antidepressants versus psychotherapy for depression: a study using hypothetical vignettes

“…Patients randomized to the choice condition are assigned to treatments based on their preferences, whereas patients randomized to the other condition are assigned to treatments based on random assignment. Although there are potential methodological and statistical disadvantages to using these designs [34,35], incorporating preferences into clinical trials may improve the external validity of studies and more accurately reflect the effectiveness of treatments when they are implemented in the community [36]. Thus far, trials conducted in depressed samples using these designs have produced mixed results [18].…”

Revisiting a non-significant findings study: A parent mentor intervention trial as exemplar

“…The result is 4 arms, with two accounting for preference. However, there are some limitations such as unequal distribution across groups, the need to power with greater numbers, increasing the cost and the potential for overestimating the treatment effect (Gemmel and Dunn 2011) It does account though for those who have a strong preference and would in the case of a parent mentor intervention have the potential to strengthen the dose and use frequency. It would also allow us to better determine the varying needs by preference group including different parenting styles, personality traits, and interpersonal skills.…”