Validity of Therapist Self-Report Ratings of Fidelity to Evidence-Based Practices for Adolescent Behavior Problems: Correspondence between Therapists and Observers

Abstract

Developing therapist-report fidelity tools to support quality delivery of evidence-based practices in usual care is a top priority for implementation science. This study tested the reliability and accuracy of two groups of community therapists who reported on their use of family therapy (FT) and motivational interviewing/cognitive-behavioral therapy (MI/CBT) interventions during routine treatment of inner-city adolescents with conduct and substance use problems. Study cases (n = 45) were randomized into two conditions: (a) Routine Family Therapy (RFT), consisting of a single site that featured family therapy as its standard of care for behavioral treatment; or (b) Treatment As Usual (TAU), consisting of five sites that featured non-family approaches. Therapists and trained observational raters provided FT and MI/CBT adherence ratings on 157 sessions (104 RFT, 53 TAU). Overall therapist reliability was adequate for averaged FT ratings (ICC = .66) but almost non-existent for MI/CBT (ICC = .06); moreover, both RFT and TAU therapists were more reliable in reporting on FT than on MI/CBT. Both groups of therapists overestimated the extent to which they implemented FT and MI/CBT interventions. Results offer support for the feasibility of using existing therapist-report methods to anchor quality assurance procedures for FT interventions in real-world settings, though not for MI/CBT.

Benchmarking Family Therapy for Adolescent Behavior Problems in Usual Care: Fidelity, Outcomes, and Therapist Performance Differences

“…Specifically, an earlier study with the current sample (Hogue, Dauber, Lichvar, et al, 2015) found that UC-FT therapists reported a significantly higher mean level of FT techniques (M = 2.4; SD = .63) than did observational coders (M = 2.0; SD = .45) for the same set of sessions, thereby inflating their scores by an average of four-tenths of a scale point. This simple algebraic adjustment has the virtue of transparency, though it can be applied only to samples that have both therapist- and observer-report fidelity data on a common metric.…”

“…Therapist-report measures can also enrich routine QA via feedback loops of several kinds, as exemplified in the current study: as a self-check by therapists to mark their own progress in treating cases; a supervision aid for trainers to monitor fidelity; and administrative data for reviewers to evaluate therapist- and agency-level performance (Schoenwald, Letourneau, & Halliday-Boykins, 2005). A previous study documented that the UC-FT therapists in the current sample were reliable in reporting on their own adherence to core FT techniques (Hogue, Dauber, Lichvar, Bobek, & Henderson, 2015; see Fidelity Measure in Method section).…”

“…The reliability of therapist-report ratings on the FT scale of the ITT-ABP was confirmed by observational coders utilizing the same 5-point rating scale (Hogue, Dauber, Lichvar, et al, 2015). Therapists and observers independently provided FT scale ratings on 157 sessions from both the UC-FT and UC-Other conditions of the parent trial.…”

“…Treatment fidelity was evaluated along with client outcomes, creating the opportunity to benchmark both inputs and outputs of behavioral treatment. Adherence to specific treatment techniques was assessed at virtually every session using an observationally validated therapist-report fidelity measure (Hogue, Dauber, Lichvar, et al, 2015), creating a dense set of fidelity data for analysis. Outcomes included dimensional measures of both internalizing and externalizing problems that are prevalent among adolescent referrals to behavioral care.…”

“…One study using therapist-report data (Hogue, Dauber, & Henderson, 2014) found that, compared to UC-Other clinicians (who all practiced non-family approaches), UC-FT clinicians reported stronger allegiance and skill in FT techniques prior to treating study cases and greater utilization of FT techniques than techniques associated with cognitive-behavioral therapy (CBT), motivational interviewing (MI), or drug counseling (DC) while treating study cases. A follow-up observational fidelity study (Hogue, Dauber, Lichvar, et al, 2015) reported that UC-FT sessions evidenced greater use of FT techniques than MI/CBT techniques, even after controlling for therapist effects. Finally, at the end of each session study therapists were asked to document that session’s format (Individual/Family versus Group) and participants (adolescent, caregiver, and/or other person); previous research (Hogue, Dauber, Henderson, & Liddle, 2014) indicates that therapists are highly reliable in documenting these structural features of treatment delivery.…”