Aims To compare characteristics of individuals participating in randomized control trials (RCTs) of treatments of substance use disorder (SUD) with individuals receiving treatment in usual care settings, and to provide a summary quantitative measure of differences between characteristics of these two groups of individuals using propensity score methods. Design Analyses using data from RCT samples from the National Institute of Drug Abuse Clinical Trials Network (CTN) and target populations of patients drawn from the Treatment Episodes Data Set-Admissions (TEDS-A). Settings Multiple clinical trial sites and nationwide usual SUD treatment settings in the United States. Participants A total of 3,592 individuals from 10 CTN samples and 1,602,226 individuals selected from TEDS-A between 2001 and 2009. Measurements The propensity scores for enrolling in the RCTs were computed based on the following nine observable characteristics: sex, race/ethnicity, age, education, employment status, marital status, admission to treatment through criminal justice, intravenous drug use, and the number of prior treatments. Findings The proportion of those with ≥12 years of education and the proportion of those who had full-time jobs were significantly higher among RCT samples than among target populations (in seven and nine trials, respectively, at p<.001). The pooled difference in the mean propensity scores between the RCTs and the target population was 1.54 standard deviations and was statistically significant at p<.001. Conclusions In the USA, individuals recruited into randomized control trials (RCT) of substance use disorder treatments appear to be very different from individuals receiving treatment in usual care settings. Notably, RCT participants tend to have more years of education and a greater likelihood of full-time work compared with people receiving care in usual care settings.

In the presence of treatment effect heterogeneity, the average treatment effect (ATE) in a randomized controlled trial (RCT) may differ from the average effect of the same treatment if applied to a target population of interest. If all treatment effect moderators are observed in the RCT and in a dataset representing the target population, we can obtain an estimate for the target population ATE by adjusting for the difference in the distribution of the moderators between the two samples. This paper considers sensitivity analyses for two situations: (1) where we cannot adjust for a specific moderator V observed in the RCT because we do not observe it in the target population; and (2) where we are concerned that the treatment effect may be moderated by factors not observed even in the RCT, which we represent as a composite moderator U . In both situations, the outcome is not observed in the target population. For situation (1), we offer three sensitivity analysis methods based on (i) an outcome model, (ii) full weighting adjustment, and (iii) partial weighting combined with an outcome model. For situation (2), we offer two sensitivity analyses based on (iv) a bias formula and (v) partial weighting combined with a bias formula. We apply methods (i) and (iii) to an example where the interest is to generalize from a smoking cessation RCT conducted with participants of alcohol/illicit drug use treatment programs to the target population of people who seek treatment for alcohol/illicit drug use in the US who are also cigarette smokers. In this case a treatment effect moderator is observed in the RCT but not in the target population dataset.

Aims To compare randomized controlled trial (RCT) sample treatment effects with the population effects of substance use disorder (SUD) treatment. Design Statistical weighting was used to re-compute the effects from ten RCTs such that the participants in the trials had characteristics that resembled those of patients in the target populations. Settings Multi-site RCTs and usual SUD treatment settings in the USA. Participants A total of 3,592 patients in ten RCTs and 1,602,226 patients from usual SUD treatment settings between 2001 and 2009. Measurements Three outcomes of SUD treatment were examined: retention, urine toxicology, and abstinence. We weighted the RCT sample treatment effects using propensity scores representing the conditional probability of participating in RCTs. Findings Weighting the samples changed the significance of estimated sample treatment effects. Most commonly, positive effects of trials became statistically non-significant after weighting (three trials for retention and urine toxicology, and one trial for abstinence); but also, non-significant effects became significantly positive (one trial for abstinence), and significantly negative effects became non-significant (two trials for abstinence). There was suggestive evidence of treatment effect heterogeneity in subgroups that are under- or over-represented in the trials, some of which were consistent with the differences in average treatment effects between weighted and unweighted results. Conclusions The findings of randomized controlled trials (RCTs) for substance use disorder treatment do not appear to be directly generalizable to target populations when the RCT samples do not adequately reflect the target populations and there is treatment effect heterogeneity across patient subgroups.

Background and Aims Global payment and accountable care reform efforts in the US may connect more individuals with substance use disorders (SUD) to treatment. We tested whether such changes instituted under an ‘Alternative Quality Contract’ (AQC) model within the Blue Cross Blue Shield of Massachusetts’ (BCBSMA) insurer increased care for individuals with SUD. Design Difference-in-differences design comparing enrollees in AQC organizations with a comparison group of enrollees in organizations not participating in the AQC. Setting Massachusetts, USA. Participants BCBSMA enrollees aged 13–64 from 2006–2011 (three years prior to and after implementation) representing 1,333,534 enrollees and 42,801 SUD service users. Measurements Outcomes were SUD service use and spending and SUD performance metrics. Primary exposures were enrollment in an AQC provider organization and whether the AQC organization did or did not face risk for behavioral health costs. Findings Enrollees in AQC organizations facing behavioral health risk experienced no change in the probability of using SUD services (1.64% vs. 1.66%; p=0.63), SUD spending ($2,807 vs. $2,700; p=0.34) or total spending ($12,631 vs. $12,849; p=0.53), or SUD performance metrics (identification: 1.73% vs. 1.76%, p=0.57; initiation: 27.86% vs. 27.02%, p=0.50; engagement: 11.19% vs. 10.97%, p=0.79). Enrollees in AQC organizations not at risk for behavioral health spending experienced a small increase in the probability of using SUD services (1.83% vs. 1.66%; p=0.003) and the identification performance metric (1.92% vs. 1.76%; p=0.007), and a reduction in SUD medication use (11.84% vs. 14.03%; p=0.03) and the initiation performance metric (23.76% vs. 27.02%; p=0.005). Conclusions A global payment and accountable care model introduced in Massachusetts USA (in which a health insurer provided care providers with fixed prepayments to cover most or all of their patients’ care during a specified time period, incentivizing providers to keep their patients’ healthy and reduce costs) did not lead to sizable changes in substance use disorder service use during the first three years following its implementation.