The influence of student characteristics on the dependability of behavioral observation data.

Abstract

Although generalizability theory has been used increasingly in recent years to investigate the dependability of behavioral estimates, many of these studies have relied on use of general education populations as opposed to those students who are most likely to be referred for assessment due to problematic classroom behavior (e.g., inattention, disruption). The current study investigated the degree to which differences exist in terms of the magnitude of both variance component estimates and dependability coefficients between students nominated by their teachers for Tier 2 interventions due to classroom behavior problems and a general classroom sample (i.e., including both nominated and non-nominated students). The academic engagement levels of 16 (8 nominated, 8 non-nominated) middle school students were measured by 4 trained observers using momentary time-sampling procedures. A series of G and D studies were then conducted to determine whether the 2 groups were similar in terms of the (a) distribution of rating variance and (b) number of observations needed to achieve an adequate level of dependability. Results suggested that the behavior of students in the teacher-nominated group fluctuated more across time and that roughly twice as many observations would therefore be required to yield similar levels of dependability compared with the combined group. These findings highlight the importance of constructing samples of students that are comparable to those students with whom the measurement method is likely to be applied when conducting psychometric investigations of behavioral assessment tools.

Psychometric Considerations for Conducting Observations Using Time-Sampling Procedures

“…Although some of the variability in regard to the number of observations needed to obtain reliable/dependable data are likely attributable to differences in the time-sampling method utilized as well as the length of observation, it is important to note that the characteristics of the students varied across studies as well. Prior research has highlighted the fact that a greater number of observations may be needed to dependably assess the behavior of those students referred for behavioral concerns than their typical peers (Briesch, Volpe, & Ferguson, 2013). Whereas all of the students in the study by Volpe et al (2009) had been referred for concerns related to their behavioral-emotional functioning, the other three studies focused on students within general education classrooms.…”

Psychometric Properties of the Behavior Assessment System for Children Student Observation System (BASC-3 SOS) with Young Children in Special Education

“…Teacher rating scales that represent constructs such as hyperactivity, inattention, and aggression (e.g., Behavior Assessment System for Children–Third Edition [BASC-3; Reynolds & Kamphaus, 2015 ], Sutter–Eyberg Student Behavior Inventory, Revised [SESBI-R; Eyberg & Pincus, 1999 ]) are another option. A strength of rating scales is strong evidence for reliability and validity of many scales (e.g., SESBI-R: Querido & Eyberg, 2003 ), while an important limitation is that they represent a teacher’s perspective and thus may be biased (Briesch et al, 2018 ). Another approach is direct behavior ratings (DBR) of specific target behaviors over brief time periods, such as a classroom activity.…”

“…SDOs are typically conducted over 10–30 min and can quantify behavior in terms of features such as frequency, duration, and time sampling interval recording (e.g., partial interval, in which the behavior is coded if it occurs at least once in the interval; whole interval, in which the behavior is coded only if it occurs for the duration of the interval; and momentary time sampling, in which the behavior is coded if it occurs at a designated point in the interval). While training to use SDOs can be time consuming and SDOs only capture a brief sample of behavior, SDO data are more objective than teacher ratings, DBRs, or narrative observations (Briesch et al, 2018 ).…”

“…There are a variety of SDOs for measuring classroom behavior of school-age children, some of which have strong psychometric properties (see Briesch et al, 2018 for review). However, there are far fewer SDOs developed or recommended for children 5 years and younger.…”

Measurement Invariance of a Direct Behavior Rating Multi Item Scale across Occasions

“…Previous studies found that DBR generates reliable scores by reflecting a large amount of variance explained by the actual student's behavior (e.g., Owens and Evans 2017). However, results from different raters across multiple time points indicate that different persons rate the same behavior differently and that students behave differently across multiple occasions (e.g., Briesch et al 2010;Volpe and Briesch 2012;Briesch et al 2014). Therefore, multiple measurement points are necessary to provide a stable score that still is interpretable.…”