Practice makes proficient: Evaluation of implementation fidelity following COMPASS consultation training

Abstract

A training package for the Collaborative Model for Promoting Competence and Success (COMPASS), a multilevel consultation and coaching intervention for improved educational outcomes of students with autism, was developed and evaluated. Using a Hybrid Type 3 design with emphasis on implementation and a multidimensional approach to evaluating implementation outcomes, we tested the training package with community-based consultant trainees (CTs) unfamiliar with COMPASS and evaluated acceptance, appropriateness, feasibility, and fidelity from multiple sources (trainees, teachers, and parents). Results confirm that COMPASS-naïve CTs can be successfully trained. At least one feedback session was needed to achieve proficiency. Initial fidelity ratings between researchers and CTs were disparate suggesting self-report may not be adequate. Four feedback opportunities were required to achieve proficiency in writing intervention plans, an activity particularly challenging for CTs. Teachers and parents perceived COMPASS as acceptable, appropriate, and feasible. CTs knowledge of evidence-based practices (EBPs) increased significantly following training; however, positive attitudes toward EBPs did not. The implementation outcomes suggest that the training package was effective for training CTs; however, additional practice with writing intervention plans is warranted.

Type and Dosage of Performance Feedback Following COMPASS Consultation on Teacher and Student Outcomes

“…The two in-person training sessions were spaced roughly a month apart to give CTs the opportunity to conduct at least one initial consultation prior to the coaching/emailed feedback training. CTs received supervision from the researchers following each consultation, coaching session, and emailed feedback report focusing on the CTs' fidelity to implementation, quality of delivery, and teacher/caregiver acceptance (Ruble et al, 2022). In total, CTs received approximately 28 hours of direct training by the researchers in COMPASS consisting of 12 hours of in-person training, approximately 12 hours of online self-directed training, and four 1-hour supervision sessions (two consultation and two coaching), followed by additional supervision reports for each consultation and coaching session conducted by the CT. Ruble et al (2022) present the implementation outcomes of the COMPASS training package in further detail.…”

“…CTs received supervision from the researchers following each consultation, coaching session, and emailed feedback report focusing on the CTs' fidelity to implementation, quality of delivery, and teacher/caregiver acceptance (Ruble et al, 2022). In total, CTs received approximately 28 hours of direct training by the researchers in COMPASS consisting of 12 hours of in-person training, approximately 12 hours of online self-directed training, and four 1-hour supervision sessions (two consultation and two coaching), followed by additional supervision reports for each consultation and coaching session conducted by the CT. Ruble et al (2022) present the implementation outcomes of the COMPASS training package in further detail. Overall, the training package was found to be effective in training CTs to implement COMPASS with high fidelity and high teacher/caregiver responsiveness (Ruble et al, 2021(Ruble et al, , 2022.…”

“…The remaining time focuses on the collaborative development of intervention plans for each goal with a detailed step-by-step teaching plan utilizing a common elements approach for instruction (Ruble et al, 2021) and the appropriate EBPs related to the goal (AFIRM, Sam et al, 2020; https://afirm. fpg.unc.edu/node/137) adapted to the child's strengths and needs (Ruble et al, 2022).…”

“…The results showed that coaching resulted in teachers not only using more EBPs but also implementing those EBPs with higher implementation fidelity, which in turn resulted in higher student goal-attainment outcomes than for students whose teachers had implemented services as usual (Sam et al, 2021). An example illustrating the latter broad area, Ruble et al (2010Ruble et al ( , 2012Ruble et al ( , 2013aRuble et al ( , 2015Ruble et al ( , 2018Ruble et al ( , 2022 conducted a series of randomized controlled trials (RCTs) examining a personalized parent-teacher goal setting, intervention planning, and coaching intervention based on shared decision-making called Collaborative Model for Promoting Competence and Success (COMPASS). Rather than focusing on the implementation of focused EBPs, COMPASS coaching focuses on teacher implementation of individualized teaching plans guided by selected EBPs personalized for the specific needs and resources of the student (Ruble et al, 2022) together with an approach that incorporates high-leverage practices (McLeskey et al, 2017) and applies principles common to high-quality intervention plans (Ruble et al, 2020(Ruble et al, , 2022.…”

“…An example illustrating the latter broad area, Ruble et al (2010Ruble et al ( , 2012Ruble et al ( , 2013aRuble et al ( , 2015Ruble et al ( , 2018Ruble et al ( , 2022 conducted a series of randomized controlled trials (RCTs) examining a personalized parent-teacher goal setting, intervention planning, and coaching intervention based on shared decision-making called Collaborative Model for Promoting Competence and Success (COMPASS). Rather than focusing on the implementation of focused EBPs, COMPASS coaching focuses on teacher implementation of individualized teaching plans guided by selected EBPs personalized for the specific needs and resources of the student (Ruble et al, 2022) together with an approach that incorporates high-leverage practices (McLeskey et al, 2017) and applies principles common to high-quality intervention plans (Ruble et al, 2020(Ruble et al, , 2022. The distinction between the content of the two different approaches is that the former addresses PF as part of a focused EBP while the latter provides PF of multiple adapted EBPs to the specific student and their situation (Ruble et al, 2020(Ruble et al, , 2022.…”