Measuring sustainment of an evidence based treatment for adolescent substance use

Abstract

Background Measuring the extent a behavioral treatment is sustained in usual care practice settings after initial support for implementation ends is challenging. Oftentimes dichotomous self-report measures are used to measure health program sustainment, but these measures may not adequately capture the quality or extent of behavioral treatment delivery. Using data collected from community based organizations that received support to implement the Adolescent Community Reinforcement Approach (A-CRA), a measure of the extent of sustainment was derived. Methods Based on implementation theories and implementation support protocols, a total of 10 core treatment elements were identified to measure the extent of sustainment using information collected from key clinical staff. Item response theory (IRT) and principal component analyses (PCA) were used to further refine the 10 elements into composite measures of sustainment. The association between the 10 elements and the relationships between the comprehensive measures to a dichotomous sustainment measure were also examined. Results Results from PCA identified two components from the initial 10 elements for measuring the extent of A-CRA treatment sustainment. The two components described different aspects of organizational support for A-CRA treatment implementation, one representing the quality or extent of treatment staffing and delivery and the other representing the quality of or extent of clinical supervision. Conclusions Using IRT and PCA, we were able to derive components that could be used to measure the extent of EBT sustainment and also better capture the quality of treatment delivery than the use of simple dichotomous measure. The methodological contribution of our study is that we have demonstrated a general analytic approach that may be applicable for other psychosocial treatments.

Comparing two federal financing strategies on penetration and sustainment of the adolescent community reinforcement approach for substance use disorders: protocol for a mixed-method study

“…Sustainment We define A-CRA sustainment over the 5-year post-grant period using a 10-element measure developed in our previous research [68], which captures the ongoing quality of treatment delivery, staffing, and supervision (see Table 3). These elements are pragmatic [75] to collect from treatment organizations and center on program-level processes, rather than clinical outcomes, as recommended for sustainment measures [76].…”

“…Each element will be assessed as follows: (1) clinician knowledge via a 10-item multiple choice survey, (2) supervisor knowledge using a 15-item true/false questionnaire, presence of certified (3) clinicians(s) and ( 4) supervisor(s) is self-reported and verified with CHS records, (5) usage by the self-reported percent of eligible youth receiving A-CRA in the past six months, (6) dosage by clinicians' self-report of the number of sessions delivered (12-14 prescribed), clinicians also report whether clinical supervision (7) occurs biweekly, (8) involves six key activities, and (9) includes a review of recorded sessions, and (10) quality of A-CRA training plans submitted by supervisors will be rated by CHS staff (i.e., percentage of expected training components included). A composite A-CRA sustainment score is created by normalizing each element (on a 0 to 1 scale, averaged across all ratings from the same treatment organization), then summing the normalized scores (i.e., composite scores range from 0 [no] to 10 [complete] sustainment) [68].…”

“…State administrators will also be asked about the infrastructure (e.g., training, policy development) developed during their grant period and plans to sustain those activities. Clinicians and supervisors will also be asked questions from our 10-element composite measure of A-CRA sustainment [68] (specified in Table 3). For organizations that did not sustain A-CRA, some questions will be anchored to the 6-month period prior to A-CRA discontinuation to allow comparable information to be collected.…”

Seeking to Understand

“…The NPD educator was the “go-to” person for questions about the intervention. The presence of clinical practice and quality improvement experts influenced sustainability for Huang et al 20 and VanDeusen Lukas et al 21 Experts coaching nurses in clinical reasoning and process improvement may positively influence their ability to adapt an EBP intervention over time effectively. The intervention should be viewed as a shared process among many stakeholders where there is ongoing experimentation and analysis of the intervention in the practice setting and ecological system 2 .…”

Assessing long-term sustainment of clinic participation in NIATx200: Results and a new methodological approach

“…Methodological approaches assessing sustainment of change are not well represented in the literature. One of these studies examined staff perspectives associated with quality implementation of an adolescent evidence-based practice (Huang et al, 2017). Another study utilized 19 quarters of administrative data to assess how organizational and therapist characteristics influenced sustainment of six different prevention-focused evidence-based practices (Brookman-Frazee et al, 2016).…”