Impact of Referral Source and Study Applicants’ Preference for Randomly Assigned Service on Research Enrollment, Service Engagement, and Evaluative Outcomes

Abstract

Objective-The inability to blind research participants to their experimental conditions is the Achilles' heel of mental health services research. When one experimental condition receives more disappointed participants, or more satisfied participants, research findings can be biased in spite of random assignment. The authors explored the potential for research participants' preference for one experimental program over another to compromise the generalizability and validity of randomized controlled service evaluations as well as cross-study comparisons.Method-Three Cox regression analyses measured the impact of applicants' service assignment preference on research project enrollment, engagement in assigned services, and a service-related outcome, competitive employment.Results-A stated service preference, referral by an agency with a low level of continuity in outpatient care, and willingness to switch from current services were significant positive predictors of research enrollment. Match to service assignment preference was a significant positive predictor of service engagement, and mismatch to assignment preference was a significant negative predictor of both service engagement and employment outcome.Conclusions-Referral source type and service assignment preference should be routinely measured and statistically controlled for in all studies of mental health service effectiveness to provide a sound empirical base for evidence-based practice.Participants in research on mental health services are rarely blind to their experimental assignments, and so attrition caused by disappointment in service assignment is a wellrecognized risk. Researchers can minimize overall attrition, as well as the threat of differential attrition (1), by screening out applicants with a priori service preferences. However, some applicants may choose not to disclose their preference to ensure that they have a chance to be assigned to a favored program, or they may feel they do not have a personal preference over and above the obvious one posed by the study design. For instance, when the experimental comparison is between a new intervention and services as usual, many applicants may prefer the new program even before they learn much about it.Unfortunately, random assignment will not always equalize preexisting study enrollee characteristics across experimental groups (2), and this will always be the case when study NIH-PA Author ManuscriptNIH-PA Author Manuscript NIH-PA Author Manuscript enrollees tend to prefer one service condition over another. Common sense tells us that service assignment preference must first be balanced within the total enrollee group for it to be distributed equitably across experimental conditions. For example, if 60% of enrollees have a preference for condition A and 40% have a preference for condition B, then, with true equivalence across conditions, service A would have 60% pleased and 40% disappointed assignees, while service B would have 40% pleased and 60% disappointed assignees. To the extent that pa...

Examining the influence of treatment preferences on attrition, adherence and outcomes: a protocol for a two-stage partially randomized trial

“…The results of five individual studies were consistent in showing that assignment to treatment, matched to participants’ preferences, is associated with enhanced adherence operationalized as attendance at treatment sessions [9, 12–14] or engagement in treatment activities [15]. In contrast, the findings of the two systematic reviews did not support the relationship between preferences and adherence [16, 17].…”

Assessment of preferences for treatment: Validation of a measure

“…Researchers have been unable to identify a consistent set of factors that significantly predict preferences. In experimental studies researchers examined the effects of treatment preferences on recruitment and retention of participants, adherence to treatment, and achievement of outcomes (e.g., Adamson, Sellman, & Dore, 2005; Awad, Shapiro, Lund, & Feine, 2000; Klaber-Moffett et al, 2004; Macias et al, 2005). Similarly, the results have been inconsistent, indicating that preferences for treatment have either no or minimal influence on recruitment, attrition, adherence to treatment, and outcomes.…”

When Programs Benefit Some People More than Others: Tests of Differential Service Effectiveness

“…To control for participants’ pre-existing attitudes toward either experimental program (Macias et al 2005), we recorded each applicant’s program preference at the time of application, and then recoded these preferences as match and mismatch to preference versus no prior preference following randomization to experimental conditions.…”