Comparison of Observational Methods and Their Relation to Ratings of Engagement in Young Children

Abstract

Although, collectively, results of earlier direct observation studies suggest momentary time sampling (MTS) may offer certain technical advantages over whole-interval (WIR) and partial-interval (PIR) recording, no study has compared these methods for measuring engagement in young children in naturalistic environments. This study compared direct observations of child engagement using WIR, PIR, and MTS methods to continuous duration recording (CDR), using measurement error, correlations, and inter-observer agreement. Results indicated that MTS consistently demonstrated measurements closer to CDR compared with PIR and WIR, although in this study, the differences between MTS and PIR across indicators were not as great as has been demonstrated in the literature. WIR, PIR, and MTS were all strongly correlated with CDR, although there were issues with under and overestimations. Finally, the four observational methods were all moderately to strongly correlated with expert ratings of engagement, although none was significantly correlated with teacher rankings of classroom engagement.

Brief Report: Measurement of Young Children’s Engagement and Problem Behavior in Early Childhood Settings

“…The differences were notable in that higher estimates of duration of engagement and lower estimates of duration of problem behavior were obtained in CI activities when compared with TD activities consistently across all children. This study also supports previous research (Devine et al, 2011;Mann et al, 1991;Wood et al, 2015) that found PIR consistently overestimates occurrences of behavior, whereas MTS produces estimates closer to CR. Moreover, this study supports the conclusions of the Wood et al (2015) study that MTS might be a viable option for obtaining estimates of engagement and problem behaviors in early childhood settings.…”

“…The observation method chosen should consider the dimensions (e.g., frequency or duration) of the behavior being measured to ensure the sample obtained represents an estimate that is "true" to the actual occurrence of the behavior (Lewis et al, 2014). In addition, researchers and practitioners may need to consider the practicality of different observation methods to ensure that the method is feasible given the context (Wood, Hojnoski, Laracy, & Olson, 2015).…”

“…To date, the Wood et al (2015) study is the only study that has compared time sampling methods to CR when measuring young children's engagement in authentic early childhood settings (i.e., classrooms) and, to our knowledge, no study has compared direct observation methods when measuring young children's problem behaviors within these settings. Moreover, although research has shown differences in young children's engagement and problem behaviors across different classroom activities, Wood et al (2015) only examined children's engagement in TD activities. Therefore, the purpose of this study is twofold: (a) to examine differences in engagement and problem behaviors of young children in early childhood settings across two different classroom activities; and (b) to examine differences between the measures of engagement and problem behaviors obtained using CR and the estimates of those behaviors obtained using time sampling recording methods.…”

Considerations for Time Sampling Interval Durations in the Measurement of Young Children’s Classroom Engagement

“…The current study used classroom observations collected as part of a larger validation project for the BOSS-EE (Hojnoski et al, 2014). Recruitment procedures, study participants, and setting were previously described in Wood et al (2016).…”

Data-Based Decision Making for Social Behavior: Setting a Research Agenda

“…Recently, scholars have begun to examine potential behavioral progress monitoring tools that can be tailored to individual behaviors and interventions at Tiers 2 and 3. These include SDO measures such as momentary time sampling of academic engagement, as well as direct behavior ratings (DBRs) of engagement and disruption, both of which have evidence of reliability (Chafouleas et al, 2010; Wood, Hojnoski, Laracy, & Olson, 2016). The National Center on Intensive Intervention (NCII; www.intensiveintervention.org) offers a critical analysis of these various tools, including information on the technical adequacy (e.g., reliability, validity, sensitivity to change) and social validity (e.g., acceptability, feasibility, contextual fit).…”