Generalizability and Decision Studies of a Treatment Adherence Instrument

Abstract

Observational measurement of treatment adherence has long been considered the gold standard. However, little is known about either the generalizability of the scores from extant observational instruments or the sampling needed. We conducted generalizability (G) and decision (D) studies on two samples of recordings from two randomized controlled trials testing cognitive-behavioral therapy for youth anxiety in two different contexts: research versus community. Two doctoral students independently coded 543 session recordings from 52 patients treated by 13 therapists. The initial G-study demonstrated that context accounted for a disproportionately large share of variance, so we conducted G- and D-studies for the two contexts separately. Results suggested that reliable cognitive-behavioral therapy adherence studies require at least 10 sessions per patient, assuming 12 patients per therapists and two coders-a challenging threshold even in well-funded research. Implications, including the importance of evaluating alternatives to observational measurement, are discussed.

The Dialectical Behavior Therapy Adherence Coding Scale (DBT ACS): Psychometric properties.

“…The DBT ACS was also found to discriminate between therapists of differing levels of expertise. Similar to studies of other treatments (e.g., Southam-Gerow et al, 2020), research therapists in efficacy trials were found to demonstrate significantly higher adherence than community therapists in effectiveness trials. This difference would be expected given that research therapists were highly trained and required to demonstrate the ability to consistently deliver DBT with adherence, whereas therapists delivering DBT in routine practice settings vary widely in terms of training and are not held to rigorous adherence standards.…”

“…Interrater reliability could not be calculated for the Exposure subscale due to the lack of nonzero scores for the item measuring the use of formal exposure. These estimates of interrater reliability compare favorably to those found for standardized measures of adherence to other psychotherapies (e.g., Barber et al, 2003; Bjaastad et al, 2015; Segal et al, 2002; Southam-Gerow et al, 2020). Additionally, interrater reliability was excellent (ICCs = .83–.99) in four of the five studies for which these data were available and good (ICC = .70) in the one study that was restricted to adolescent patients (McCauley et al, 2018).…”

“…Based on prior research (Harned et al, 2014), we also hypothesized that the DBT ACS would have excellent interrater reliability. Consistent with adherence measures for other treatments (e.g., Southam-Gerow, et al, 2020), we hypothesized that the DBT ACS would discriminate between DBT and a different treatment as well as between research and community therapists who were expected to have different levels of expertise in DBT. Finally, we hypothesized that therapist variance in adherence would be smaller among research therapists compared to community therapists and, as a result, more sessions would need to be coded to achieve a dependable DBT ACS score among research than community therapists.…”

Research Review: Pediatric anxiety disorders – what have we learnt in the last 10 years?

“…Importantly, youth report better relationships with their CBT therapists following the introduction of exposure (Cummings et al,2013). At the same time, the gap between research and practice settings persists, and among therapists, CBT adherence and competence – key components of treatment success – are better in research versus community settings, with the latter group difference appearing particularly pronounced when exposures are introduced (Southam‐Gerow et al, 2020).…”