Toward National Estimates of Effectiveness of Treatment for Substance Use

Abstract

Objective To estimate how results would have varied if a substance abuse clinical trial had been conducted with nationally representative adults with substance use and with representative adults receiving substance use treatment. Method Results were analyzed from a multisite clinical trial comparing the effectiveness of the Therapeutic Education System (TES) to treatment-as-usual for outpatient addiction treatment (n=507). Abstinence in the last four weeks of treatment was the primary outcome. The general population sample and general population treated samples were derived from Wave 1 of the National Epidemiological Survey on Alcohol and Related Conditions (NESARC) (n=43,093). Propensity scores provided a standardized measure of the difference between clinical trial participants and the two NESARC samples. The clinical trial was reanalyzed by reweighting the sample with propensity scores derived for the two samples to obtain generalizable estimates of treatment effects. Results Prior to reweighting, the Odds Ratio (OR) of response to TES versus treatment-as-usual in the trial was 1.62 (95% CI=1.12–2.35). After reweighting the sample to be representative of the two groups, ORs were 1.33 (95% CI=0.34–5.26) for the representative sample with any substance use and 1.64 (95% CI=0.82–3.27) for the representative treated sample. Conclusion Applying propensity score weighting to clinical trial results provides a method for estimating the population generalizability of clinical trial findings that relies on effect moderators observed in the study sample and population. Broader confidence intervals in the reweighted samples do not necessarily indicate lack of efficacy of TES, but rather greater uncertainty concerning effectiveness in general population samples.

Generalizability of the findings from a randomized controlled trial of a web‐based substance use disorder intervention

“…The findings are also consistent with Susukida et al, 16 which found the significance of estimated sample treatment effects was different from that of the population effects when the distribution of characteristics of RCT samples were made to resemble the distribution of the target populations by using the same statistical weighting techniques that this study used. Especially in the context of the generalizability of the findings of a web-based SUD intervention, this study’s findings echo the findings of the recent study by Blanco et al, 17 which found that the significant treatment effect of TES estimated through RCT was insignificant after weighting the sample to resemble the target populations drawn from the 2001–2002 NESARC data. Our study confirms that their findings hold when the generalizability of the TES RCT was assessed with the target population from more recent years and with a population obtained from administrative data from usual treatment settings across the US.…”

“…However, very few previous studies assessed the representativeness of the RCT participants of web-based SUD interventions as compared with the intended target populations, and whether and how the sample representativeness impacts the generalizability of RCT findings. A recent study by Blanco et al 17 directly compared the RCT sample of the web-based intervention, the Therapeutic Education System (TES), with two target populations: individuals seeking SUD treatment and individuals with SUD regardless of their treatment seeking behavior, both drawn from the Wave 1 of the National Epidemiologic Survey on Alcohol and Related Conditions (NESARC), conducted between 2001 and 2002. These authors found substantial differences in characteristics between the RCT sample and the two target populations and also demonstrated that the treatment effect of TES became insignificant after the sample was weighted to resemble these target populations.…”

“…These authors found substantial differences in characteristics between the RCT sample and the two target populations and also demonstrated that the treatment effect of TES became insignificant after the sample was weighted to resemble these target populations. However, illicit drug use in the US has been increasing in the past 10 to 15 years and the NESARC target populations used in Blanco et al 17 may not appropriately represent the current population with SUD. Approximately 9.4 percent of the US population used an illicit drug during the past month in 2013 compared with 8.3 percent in 2002.…”

“…19 Given these recent changes in nationwide trends in substance use behaviors, it is important to assess the representativeness of the SUD RCTs compared with more current target population. Furthermore, in Blanco et al, 17 both the treatment-seeking target population and the general target population were drawn from the same source, a general population survey that did not capture marginalized population groups such as homeless and/or incarcerated individuals.…”

“…The aims of this current study were (1) to assess the sample representativeness of an RCT of a web-based TES with the target population of individuals with recent drug use admitted into SUD treatment in usual care settings, and (2) to estimate sample treatment effects and the population effects of the web-based intervention. Unlike the study by Blanco et al, 17 this study used more recent target population, which was drawn from administrative data on individuals in treatment settings, including patients referred to treatment through the judicial system and marginalized population groups.…”