Tracking Evidence Based Practice with Youth: Validity of the MATCH and Standard Manual Consultation Records

Abstract

This study sought to evaluate the agreement between therapist report and coder observation of therapy practices. The study sampled session data from a community-based, randomized trial of treatment for youth ages 7 to 13. We used therapist report of session content and coverage gathered using formal Consultation Records and developed complimentary records for coders to use when watching or listening to therapy tape. We established initial reliability between coders and then conducted a random, stratified, and comprehensive sample of sessions across youth (N = 121), therapists (N = 57), conditions (MATCH and Standard Manuals), and study sites (Honolulu and Boston) to code and compare with therapist record reports. Intraclass correlation coefficients (ICCs) representing coder versus therapist agreement on manual content delivered ranged from .42 to 1.0 across conditions and problem areas. Analyses revealed marked variability in agreement regarding whether behavioral rehearsals took place (ICCs from -.01 to 1.0) but strong agreement on client comprehension of therapy content and homework assignments. Overall, the findings indicate that therapists can be accurate reporters of the therapeutic practices they deliver, although they may need more support in reporting subtle but valuable aspects of implementation such as types of behavioral rehearsals. Developing means to support accurate reporting is important to developing future clinical feedback methodology applicable to the implementation of evidence-based treatments in the real world.

A Digital Feedback System to Support Implementation of Measurement-Based Care by School-Based Mental Health Clinicians

“…Second, MBC practices were evaluated using clinician self-report, which has been criticized as potentially biased relative to “gold standard” practice observation (Hurlburt, Garland, Nguyen, & Brookman-Frazee, 2009; Miller & Mount, 2001; Nakamura et al, 2014). Nevertheless, multiple studies have also found significant, positive relationships between clinician self-report and observational assessments, especially for specific practices that, like MBC, are less complicated than full treatment protocols (Chapman, McCart, Letourneau, & Sheidow, 2013; Hogue, Dauber, Henderson, & Liddle, 2013; Ward et al, 2013). Furthermore, the current study was primarily focused on documenting changes in MBC practice between groups and over time, rather than interpreting absolute rates of assessment administration and feedback.…”

Validity of Therapist Self-Report Ratings of Fidelity to Evidence-Based Practices for Adolescent Behavior Problems: Correspondence between Therapists and Observers

“…Interestingly, novice family therapists showed greater agreement with raters than did veterans, perhaps because they were more conservative overall in rating themselves, although the small subgroup sizes preclude confident generalizations. Given that study therapists received only a single training session in using the self-report coding system, there is reason to believe that their reliability could be substantially improved if ongoing training and feedback in reliable documentation of FT adherence were to be incorporated into routine QA procedures (Hogue et al 2013; Ward et al 2013). …”

“…Thus there is concordance between the design of the instrument and the (lack of) of standardization among the “loose confederation” of interventions that it measures—it is intended to identify a generic approach, not a specific model. In contrast, the fidelity tools used by therapists implementing the modular MATCH-ADC protocol (Weisz et al 2012) were technically elaborate and procedure-driven (Ward et al 2013), matching the extensive standardization of the protocol itself. These two examples underscore a general principle: Fidelity measures and procedures can and should be calibrated according to the degree of specificity, flexibility, and customization needed to effectively capture the treatment constructs of interest and meet the desired evaluation goals (Regan et al 2013).…”

A Statewide Common Elements Initiative for Children’s Mental Health

“…61 Regardless, determining whether such changes have occurred may require more rigorous and direct methods. The inclusion of consultant ratings, although more rigorous than clinician self-report, still relied on assessing clinician self-report of their sessions with clients.…”