Measuring treatment differentiation for implementation research: The Therapy Process Observational Coding System for Child Psychotherapy Revised Strategies Scale.

Abstract

Observational measures to assess implementation integrity (the extent to which components of an evidence-based treatment are delivered as intended) are needed. We evaluated the reliability of the scores and the validity of the score interpretations for the Therapy Process Observational Coding System for Child Psychotherapy – Revised Strategies scale (TPOCS-RS; McLeod, 2010) and assessed the potential of the TPOCS-RS to assess treatment differentiation, a component of implementation integrity. The TPOCS-RS includes five theory-based subscales (Cognitive, Behavioral, Psychodynamic, Client-Centered, Family). Using the TPOCS-RS, coders independently rated 954 sessions conducted with 89 children (M age = 10.56, SD = 2.00; aged 7–15 years; 65.20% Caucasian) diagnosed with a primary anxiety disorder who received different treatments (manual-based vs. non-manualized) across settings (research vs. practice). Coders produced reliable ratings at the item level (M ICC = .76, SD = .18). Analyses support the construct validity of the Cognitive and Behavioral subscale scores and, to a lesser extent, the Psychodynamic, Family, and Client-Centered subscale scores. Correlations among the TPOCS-RS subscale scores and between the TPOCS-RS subscale scores and observational ratings of the alliance and client involvement were moderate suggesting independence of the subscale scores. Moreover, the TPOCS-RS showed promise for assessing implementation integrity as the TPOCS-RS subscale scores, as hypothesized, discriminated between manual-guided treatment delivered across research and practice settings and non-manualized usual care. The findings support the potential of the TPOCS-RS Cognitive and Behavioral subscales to assess treatment differentiation in implementation research. Results for the remaining subscales are promising, although further research is needed.

Behavioral Rehearsal for Analogue Fidelity: Feasibility in a State-Funded Children’s Mental Health Initiative

“…In contrast to Model Explanation, very few clinicians achieved acceptable levels of analogue fidelity at either time point for Homework Planning. These rates are particularly disappointing, but are not entirely out of line with other studies of technique use among EBT-trained clinicians in public mental health settings (McLeod, Smith, Southam-Gerow, Weisz, & Kendall, 2015). …”

Does the Delivery of CBT for Youth Anxiety Differ Across Research and Practice Settings?

“…A coder considers both thoroughness and frequency while making a rating; extensiveness ratings thus provide dosage information about each intervention. Previous studies have demonstrated that the TPOCS-RS, or variants thereof (i.e., PRAC-TPOCS, Garland et al, 2010; TPOCS-S, McLeod & Weisz, 2010), have demonstrated item inter-rater reliability ranging from .71 to .86 ( M ICC = .81), the item and subscale scores provide evidence of construct validity across research and practice settings (McLeod & Weisz, 2010; McLeod et al, 2015; Southam-Gerow et al, 2016; Wood et al, 2006), subscales scores differentiate between treatment types (McLeod et al, 2015; Southam-Gerow et al, 2010; Weisz et al, 2009; Wood et al, 2006), and subscale scores demonstrated predictive validity (Garland et al, 2014). The current study used the TPOCS-RS Psychodynamic, Family, and Client-centered subscales.…”

“…Without consensus on how to score integrity instruments (see McLeod et al, 2015; Southam-Gerow et al, 2016), we considered three approaches: (a) Average of all items on each subscale for each session (Coping Cat M = 2.63, SD = 1.22; Psychodynamic M = 1.19, SD = 0.35; Family M = 1.57, SD = 0.85; Client-Centered M = 2.76, SD = 0.76); (b) Average of all scored items (items scored above a 1) for each session (Coping Cat M = 3.08, SD = 1.22; Psychodynamic M = 1.40, SD = 0.60; Family M = 1.96, SD = 1.14; Client-Centered M = 3.11, SD = 0.79); and (c) Highest item scored for each session (Coping Cat M = 4.56, SD = 2.10; Psychodynamic M = 1.49, SD = 0.81; Family M = 2.52, SD = 1.91; Client-Centered M = 4.28, SD = 1.41). Though scores on the three scoring approaches differed in magnitude, the three scores for each subscale were highly correlated (all r s > .78).…”

Development and Initial Psychometrics for a Therapist Competence Instrument for CBT for Youth Anxiety

“…(TPOCS-RS; McLeod, Smith, Southam-Gerow, Weisz, & Kendall, 2015)is a 42-item instrument designed to assess therapist delivery of interventions across five theory-based subscales: Cognitive (4 items; e.g., Cognitive Distortions), Behavioral (9 items; e.g., Operant Interventions), Psychodynamic (5 items; e.g., Interpretation), Family (7 items; e.g., Parenting Intervention), and Client-Centered (4 items; e.g., Positive Regard).There are 13additional items (e.g., Homework, Play Therapy) that represent interventions that play a meaningful role in treatment but are not associated with a specific theory-based subscale. Coders rate the extent to which the therapist engages in each item during an entire session on a 7-point extensiveness scale(Hogue et al, 1996): 1 = not at all , 3 = somewhat , 5 = considerably , 7 = extensively .…”