Measuring Rater Reliability on a Special Education Observation Tool

Abstract

This study used generalizability theory to measure reliability on the Recognizing Effective Special Education Teachers (RESET) observation tool designed to evaluate special education teacher effectiveness. At the time of this study, the RESET tool included three evidence-based instructional practices (direct, explicit instruction; whole-group instruction; and discrete trial teaching) as the basis for special education teacher evaluation. Five raters participated in two sessions to evaluate special education classroom instruction collected from two school years, via the Teachscape 360-degree video system. Data collected from raters were analyzed in a two-facet “partially” nested design where occasions (o) were nested within teachers (t), o:t, and crossed with raters (r), {o:t} x r. Results from this study are in alignment with similar studies that found multiple observations and multiple raters are critical for ensuring acceptable levels of measurement score reliability. Considerations for the feasibility of practice should be observed in future reliability and validity studies on the RESET tool, and further work is needed to address the lack of research on rater reliability issues within special education teacher evaluation.

Evaluating Special Educators’ Classroom Performance: Does Rater “Type” Matter?

“…Larger research studies, like those included in the Measures of Effective Teaching (MET) project, may include school administrators as raters but fail to include teachers of SWDs in the samples. Studies that do include special education teachers (e.g., Semmelroth & Johnson, 2013) have included peer raters but have not utilized school administrators as raters of the teachers' instruction. Understanding rater characteristics and the conditions surrounding scoring is critical to ensuring that all teachers receive fair evaluations, but it is even more critical for special education teachers, who are often evaluated by administrators who may not possess formal training or have expertise in the area of special education (Sledge & Pazey, 2013).…”

“…The special education teacher participants from Idaho were selected from an existing database of 21 teachers who had contributed video data files of their teaching for a previous study (Semmelroth & Johnson, 2013), and provided consent for their video data to be used in future research. The video data files for the Idaho participants were captured using the Teachscape video capture system and stored in the Teachscape secure online database.…”

“…Peer raters. Peer rater data for this study were obtained from a dataset used in a prior project investigating rater reliability (Semmelroth & Johnson, 2013). As previously mentioned, 21 special education teachers in Idaho contributed video-recorded lessons; five additional special education teachers in Idaho were recruited to score the video-recorded lessons of their peers on a variety of dimensions.…”

How Much Is Enough? Evaluating Intervention Implementation Efficiently

“…Most empirically based observation tools were developed for monitoring implementation in general education (e.g., Danielson, 2013; La Paro, Pianta, & Stuhlman, 2004) and may not measure qualities of instruction known to be essential in intervention (Holdheide, Goe, Croft, & Reschly, 2010; Johnson & Semmelroth, 2012; Semmelroth & Johnson, 2014). Not only are most current tools not designed for intervention, as Knight (2007) noted, finding the personnel and time for monitoring implementation is also a challenge.…”

Using Generalizability Theory to Examine the Dependability of Scores From the Learning Target Rating Scale

“…Zaslow et al, 2011). Semmelroth and Johnson (2014) noted that efforts focused on developing and applying these types of measures of effective teaching have often 669924T ECXXX10.1177/0271121416669924Topics in Early Childhood Special EducationMcLaughlin et al…”