Variance and Reliability in Special Educator Observation Rubrics

Abstract

This study describes the development and initial psychometric evaluation of the Recognizing Effective Special Education Teachers (RESET) observation instrument. The study uses generalizability theory to compare two versions of a rubric, one with general descriptors of performance levels and one with item-specific descriptors of performance levels, for evaluating special education teacher implementation of explicit instruction. Eight raters (four for each version of the rubric) viewed and scored videos of explicit instruction in intervention settings. The data from each rubric were analyzed with a four facet, crossed, mixed-model design to estimate the variance components and reliability indices. Results show lower unwanted sources of variance and higher reliability indices with the rubric with item-specific descriptors of performance levels. Contributions to the fields of intervention and teacher evaluation are discussed.

The Relationship of Special Education Teacher Performance on Observation Instruments With Student Outcomes

“…In studies to validate content-specific rubrics in reading and math, our results have reported teacher ability centered at −.35 logits (range −1.08 to .38) on a reading comprehension rubric (E. S. Johnson, Moylan, Crawford, & Zheng, 2019), and teacher ability centered at −.05 logits (range −1.73 to 1.37) on a math instruction rubric (Crawford et al, 2019). A direct comparison is not currently possible, as these studies involved different teachers and different raters, but further research investigating special education teachers’ scores using an EI rubric as compared to a content-specific rubric in reading or math, and examining the relationship between teacher performance and student outcomes with the various teacher observation instruments could provide useful information about which protocols provide the most relevant information about a teacher’s performance and the subsequent impact on student outcomes.…”

“…Research in teacher observation has shown that teacher performance can vary depending on the time of year and based on the students in class (Mantzicopoulos et al, 2018). However, several studies have reported that between 2 and 4 observations of teachers’ instruction result in reliable estimates of teacher performance (Crawford et al, 2019; Kane & Staiger, 2012). Our findings found very little variability in teacher performance as a result of the lesson suggesting that this may not have been an issue in the current data set.…”

“…Each item is rated on a 3-point scale (1 = Not implemented , 2 = Partially implemented , and 3 = Implemented ) to evaluate a teacher’s level of proficiency in implementing that specific element. A number of studies have demonstrated that the EI protocol provides reliable assessments of a teacher’s ability to implement this EBP (Crawford et al, 2019; E. S. Johnson et al, 2018, 2019).…”

“…The EI observation protocol is one of the 21 EBPs included within the RESET system, comprised of 25 items that detail the elements of EI (E. S. Johnson, Zheng, Crawford, & Moylan, 2019) and has been found to result in reliable evaluations of teacher practice across several studies (Crawford et al, 2019; E. S. Johnson, Zheng, et al, 2018, 2019).…”

“…RESET consists of 21 observation protocols aligned with EBPs for students with high incidence disabilities, including learning disabilities. The EI observation protocol is one of the 21 EBPs included within the RESET system, comprised of 25 items that detail the elements of EI (E. S. and has been found to result in reliable evaluations of teacher practice across several studies (Crawford et al, 2019;E. S. Johnson, Zheng, et al, 2018, 2019.…”