An evaluation of observational methods for measuring response to classwide intervention.

Abstract

Although there is much research to support the effectiveness of classwide interventions aimed at improving student engagement, there is also a great deal of variability in terms of how response to group-level intervention has been measured. The unfortunate consequence of this procedural variability is that it is difficult to determine whether differences in obtained results across studies are attributable to the way in which behavior was measured or actual intervention effectiveness. The purpose of this study was to comparatively evaluate the most commonly used observational methods for monitoring the effects of classwide interventions in terms of the degree to which obtained data represented actual behavior. The 5 most common sampling methods were identified and evaluated against a criterion generated by averaging across observations conducted on 14 students in one seventh-grade classroom. Results suggested that the best approximation of mean student engagement was obtained by observing a different student during each consecutive 15-s interval whereas observing an entire group of students during each interval underestimated the mean level of behavior within a phase and the degree of behavior change across phases. In contrast, when observations were restricted to the 3 students with the lowest levels of engagement, data revealed greater variability in engagement across baseline sessions and suggested a more notable change in student behavior subsequent to intervention implementation.

Comparing Versions of the Good Behavior Game

“…The student variables remained the same, but the system was modified to monitor classwide behavior. Rather than observing one student and a peer comparison each fifth interval, a different student was observed each interval using what has been described as an individual-fixed approach (Briesch, Hemphill, Volpe, & Daniels, 2015). Observers coded the behavior of the first student and moved to the next student at the end of the interval.…”

“…This pattern continued until all students were observed and repeated until the end of the observation. Researchers found this approach to data collection closely approximates actual student behavior (Briesch et al, 2015). When two observers were coding for inter-observer agreement (IOA), before they began the observation, they discussed which student they would observe first as well as the pattern they would follow (i.e., row by row, front to back).…”

Promoting teachers’ implementation of culturally and contextually relevant class‐wide behavior plans

“…Data collectors also used a researcher-derived measure of class-wide academic engagement and disruptive behavior using a momentary time sampling procedure with 15-second intervals for the entire observation (Ferguson, Briesch, Volpe, & Daniels, 2012). Data collectors employed an individual-random observation scheme, where observers randomly selected a different student in the class to observe during each consecutive 15-second interval (Briesch, Hemphill, Volpe, & Daniels, 2015). Such measures have been previously used in research and demonstrated to provide accurate estimates of student behavior (Chafouleas, Sanetti, Kilgus, & Maggin, 2012).…”

The Caterpillar Game: A Classroom Management System

“…Another limitation was that student disruptive behavior was sampled (i.e., for each session, the researcher randomly selected three students to observe, which might not be a representation of all students' disruptive behavior). In a recent study, Briesch, Hemphill, Volpe, and Daniels (2015) concluded that the best approximation of mean student engagement was obtained by observing a different student during each consecutive 15-second interval. The authors expressed caution when observing a few students with the most behavioral difficulties as a measure of overall classroom change, because baseline behaviors for these students were more variable and may lead to embellished conclusions regarding overall class improvements.…”

“…However, given the reluctance of schools to allow videotaping and potential reactivity, it is critical that the most accurate observational methods are utilized to assess the effectiveness of classwide intervention. In addition to the utility of the observational method, when various researchers use different procedures to determine the effectiveness of an intervention, it is difficult to determine whether different results are due to the observational procedures or the intervention (Briesch et al, 2015).…”