Direct Training to Increase Agreement Between Teachers’ and Observers’ Treatment Integrity Ratings

Abstract

Treatment integrity data are critical to making informed decisions about the effectiveness of an intervention (Durlak & DuPre, 2008). These data reflect the extent to which an intervention was fully implemented as intended by a trained interventionist, and can target various aspects of intervention delivery (Sanetti & Kratochwill, 2009). Specifically, assessment might include (a) which steps of the intervention were delivered (i.e., adherence), (b) how much of the intervention was provided (i.e., dosage), (c) how well the intervention was implemented (i.e., quality), or (d) the process by which intervention delivery occurred (i.e., exposure; Dane & Schneider, 1998; Jones, Clarke, & Power, 2008). Without assessment of treatment integrity, it is difficult to determine whether outcomes, specifically poor outcomes, are due to an ineffective intervention or an intervention that was poorly implemented (Sanetti & Kratochwill, 2009). Despite the critical importance of these data, there is a gap between how and how often these data are collected in educational research versus school-based practice.

Multi‐informant assessment of treatment integrity in the classroom

“…Given prior research on performance feedback (Solomon et al, 2012) and implementation support to improve accurate treatment fidelity (Fallon et al, 2018), we hypothesize that performance feedback will improve the agreement between researcher‐collected treatment‐fidelity data and teacher self‐report or student report, respectively.…”

“…On the basis of the research on peer‐mediated interventions (Dart et al, 2014), we expect that student reports will be consistent with researcher‐collected treatment‐fidelity data. Does as needed performance feedback improve the agreement between research‐collected treatment‐fidelity data and teacher self‐report or student report, respectively?Given prior research on performance feedback (Solomon et al, 2012) and implementation support to improve accurate treatment fidelity (Fallon et al, 2018), we hypothesize that performance feedback will improve the agreement between researcher‐collected treatment‐fidelity data and teacher self‐report or student report, respectively. Does student academic engagement remain consistent when students report teachers' implementation of the GBG and receive performance feedback?…”

“…That said, others have documented teachers can accurately self‐report (e.g., teacher self‐report of math academic intervention, Sanetti & Kratochwill, 2009). There is some evidence to suggest that providing implementation support (in this case, direct training) can increase teachers' accuracy of self‐report of an explicit instruction intervention (Fallon et al, 2018). Given these mixed findings as well as its feasibility, usage in schools, and potential for prompting higher levels of treatment integrity, more research on teacher self‐report is warranted.…”

“…The promise of teacher and student reports of treatment integrity requires consideration of implementation support to facilitate accurate data. Fallon et al (2018) documented the improvement of teachers' accuracy in reporting their implementation of explicit instruction when provided with direct training. Direct training, which involves instruction, modeling, and role‐play, was more effective in facilitating accurate responding than indirect training, which involves only instruction.…”

“…Given these barriers to school psychologists collecting treatment‐integrity data, it may be time to think about how to incorporate other stakeholders into treatment integrity assessment. It may be feasible for classroom teachers to self‐report their implementation (i.e., systematically rate their completion of intervention steps during or following implementation; Fallon, Sanetti, Chafouleas, Faggella‐Luby, & Briesch, 2018). Further, for interventions involving relatively low‐stake decisions (i.e., those that are not implemented to make decisions about individual response to intervention for special education eligibility purposes), such as class‐wide group contingencies, students may be able to support treatment‐integrity data collection.…”