Occasions and the Reliability of Classroom Observations: Alternative Conceptualizations and Methods of Analysis

Abstract

Student-teacher interactions are dynamic relationships that change and evolve over the course of a school year. Measuring classroom quality through observations that focus on these interactions presents challenges when observations are conducted throughout the school year. Variability in observed scores could reflect true changes in the quality of student-teacher interaction or simply reflect measurement error. Classroom observation protocols should be designed to minimize measurement error while allowing measureable changes in the construct of interest. Treating occasions as fixed multivariate outcomes allows true changes to be separated from random measurement error. These outcomes may also be summarized through trend score composites to reflect different types of growth over the school year. We demonstrate the use of multivariate generalizability theory to estimate reliability for trend score composites, and we compare the results to traditional methods of analysis. Reliability estimates computed for average, linear, quadratic, and cubic trend scores from 118 classrooms participating in the MyTeachingPartner study indicate that universe scores account for between 57% and 88% of observed score variance.Classroom observation protocols produce direct measures of instructional quality and studentteacher interactions. Like other types of performance assessment, these measures are subject to multiple sources of measurement error such as the selection of scoring criteria, observer severity or leniency, and occasion of observation. The most problematic source of error to handle in observation protocol design is the occasion of observation. Classrooms are dynamic, complex settings, and the quality of student-teacher interactions will vary throughout the day and over the course of the school year. Variation among scores collected on multiple occasions may be attributable to real changes in the quality of student-teacher interactions or random measurement error. Differentiating the cause of score variation over time is a matter of observation protocol design and analysis. The purpose of this article is to describe approaches for conceptualizing the influence of occasions on scores and describe the use of Correspondence should be sent to Patrick Meyer, Curry

The Reliability of Framework for Teaching Scores in Kindergarten

“…In general, studies indicate that scores do not meet acceptable reliability standards (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME], 2014) when they are based on the number of observed lessons typical in teacher evaluations (e.g., Ho & Kane, 2013; Praetorius et al, 2012). In addition, ratings of instructional practices are less stable, and therefore require more observations, than those of classroom climate (Meyer et al, 2011; Praetorius et al, 2014). Two features of this research are relevant to our present study.…”

Special Education Teacher Evaluation

“…Therefore, the use of FFT to evaluate special education instruction could lead to an evaluation that is not aligned with the research base and that endorses practices that do not lead to improved outcomes for SWD. In addition, there is a growing body of research that indicates for observations to be useful for identifying effective teachers, or simply in improving classroom quality and teacher practice, these measures must use standardized observation protocols that minimize measurement error and permit valid inferences (Meyer, Cash, & Mashburn, 2011), which is work that has yet to be completed on FFT.…”

A Multivariate Generalizability Theory Approach to College Students' Evaluation of Teaching

“…In order to effectively evaluate teachers' classroom teaching performance, a number of factors should be taken into account, such as the evaluation time, curricula, and raters' preference (e.g., Goe et al, 2008; Meyer et al, 2012; Gitomer et al, 2014). For most Chinese colleges, teaching evaluations are compulsory.…”