The Impact of Training on the Accuracy of Teacher-Completed Direct Behavior Ratings (DBRs)

Abstract

The purpose of this study was to investigate the effects of three levels of training (direct, indirect, and none) on teachers’ ability to accurately rate video of student behavior. Direct and indirect training groups received instructional sessions on direct behavior ratings (DBRs), with the direct training group receiving opportunities for practice and feedback. Each training group viewed a video of a typical classroom environment and then rated the proportion of time the student displayed disruptive behavior and academic engagement. Accuracy was examined through comparison of the teachers’ ratings with direct observation data. Teacher acceptability of using DBRs was also assessed. Overall, the results suggested that the most intensive level of training (i.e., direct) did not improve the accuracy with which teachers rated behaviors using DBRs. Limitations and future implications are discussed.

Preliminary Investigation of the Impact of a Web-Based Module on Direct Behavior Rating Accuracy

“…In addition, these preliminary findings do not support universal advantage, suggesting additional or modified training may be needed to further enhance accuracy. Although a direct comparison to prior work (e.g., Chafouleas et al, 2012a;Harrison et al, in press;LeBel et al, 2010) cannot be made given varied study conditions, resulting patterns are similar across studies in suggesting that training can improve rating accuracy, but a panacea is not likely. Exact correspondence across scores may be difficult to find, particular for medium levels of behavior.…”

“…The web-based DBR training module was created based on previously reviewed findings from work investigating the impact of various components to training (i.e., Chafouleas et al, 2012a;LeBel, Kilgus, Briesch, & Chafouleas, 2010;Schlientz, Riley-Tillman, Briesch, Walcott, & Chafouleas, 2009). The training module was computerized and was comprised of three segments (i.e., overview, modeling including frame of reference, practice, and feedback).…”

Preliminary evaluation of various training components on accuracy of Direct Behavior Ratings

“…Although there are some preliminary investigations of the impact of training on outcomes for assessment methods involving DBR and direct observations, results have been somewhat inconclusive to date. For example, some research findings on the impact of training on rating accuracy using DBR-SIS suggest that accuracy increased with more extensive training procedures (Schlientz et al, 2009), whereas other findings suggest that only a moderate degree of training is sufficient (Harrison et al, Submitted for publication;LeBel et al, 2010). However, that body of work varies substantially with regard to considerations such as included training components, targets assessed, and display rates of those targets.…”

“…Results suggested that there was not substantial improvement in rating accuracy for those teachers who received the training over those who did not. Another study compared the effects of direct, indirect, and no training procedures on the correspondence between DBR-SIS and systematic direct observation data (LeBel, Kilgus, Briesch, & Chafouleas, 2010). In that study, participants in both the direct and indirect training conditions viewed an instructional DVD that included familiarization and demonstration of DBR-SIS.…”

“…The results of a power analysis for a repeated-measures multivariate analysis of variance (MANOVA) with six groups and six repetitions suggested that in order to achieve a power of .80, and assuming an effect size of .25 (see LeBel et al, 2010) and an alpha of .05, 162 participants (i.e., 27 participants per each of the six conditions) should be recruited. As such, recruitment continued until 177 participants (i.e., undergraduates enrolled in an introductory psychology course at a university located in the southeastern United States) enrolled in the study, allowing for a small degree of over enrollment.…”