Trends in Classroom Observation Scores

Abstract

Observations and ratings of classroom teaching and interactions collected over time are susceptible to trends in both the quality of instruction and rater behavior. These trends have potential implications for inferences about teaching and for study design. We use scores on the Classroom Assessment Scoring System-Secondary (CLASS-S) protocol from 458 middle school teachers over a 2-year period to study changes over time in (a) the average quality of teaching for the population of teachers, (b) the average severity of the population of raters, and (c) the severity of individual raters. To obtain these estimates and assess them in the context of other factors that contribute to the variability in scores, we develop an augmented G study model that is broadly applicable for modeling sources of variability in classroom observation ratings data collected over time. In our data, we found that trends in teaching quality were small. Rater drift was very large during raters' initial days of observation and persisted throughout nearly 2 years of scoring. Raters did not converge to a common level of severity; using our model we estimate that variability among raters actually increases over the course of the study. Variance decompositions based on the model find that trends are a modest source of variance relative to overall rater effects, rater errors on specific lessons, and residual error. The discussion provides possible explanations for trends and rater divergence as well as implications for designs collecting ratings over time.

Emotional Bias in Classroom Observations: Within-Rater Positive Emotion Predicts Favorable Assessments of Classroom Quality

“…On average, raters assessed the overall quality of classrooms slightly above the midpoint on the 7-point scale. This is consistent with prior research that has tracked overall classroom quality scores using the CLASS over a two-year period (Casabianca et al, 2015). Intraclass correlation coefficients (ICCs) showed that the majority (89%) of variability in classroom quality was at Level 1.…”

“…The unconditional intercept model estimated an overall mean in classroom quality assessments of 4.22 (SE = .04). This is just above the midpoint on the 7-point rating scale, as was found in Study 1 and in recent longitudinal data on the CLASS (Casabianca et al, 2015). ICCs showed that the majority (93%) of variability in classroom quality was at Level 1.…”

“…Furthermore, an investigation of the role of other possible betweenrater differences, such as emotional intelligence or personality, in observational assessments is needed given the notable amount of variance they contribute to overall classroom quality ratings (i.e., 4% and 8% in Studies 1 and 2; see also Casabianca et al, 2015).…”

Building a More Complete Understanding of Teacher Evaluation Using Classroom Observations

“…It is equally vital to develop systems for periodic calibration to ensure that raters' scores continue to be consistent with expert scores Park et al, 2014). Park and colleagues (2014) found that systems developed to minimize rater bias based on the characteristics of the rater, teacher, and classroom were effective-but in the context of video observations when raters were not personally connected to the teachers.Even rigorous training and calibration systems may not be sufficient in reducing variability in scores, especially when stakes are attached (Casabianca et al, 2013(Casabianca et al, , 2015. In consequential evaluations of teachers, North Carolina observers tended to rate a disproportionate numbers of teachers just above proficiency thresholds (Barrett Crittenden-Fuller & Guthrie, 2015).…”

Creating an Indicator of K--12 Classroom Coverage of Science, Technology, Engineering, and Math (STEM) Content and Practices

“…In addition, although some studies have found that structured observation protocols can be applied with adequate reliability, others have raised questions about the consistency of ratings and the validity of scores obtained from these tools for different purposes (Harris and Sass, 2007;Henry, Murray, and Phillips, 2007;HRI, 2000;Jacob and Lefgren, 2008;Kane, Kerr, and Pianta, 2014;Learning Mathematics for Teaching Project, 2011;Medley and Coker, 1987;Piburn and Sawada, 2000;Schultz and Pecheone, 2014;Walkington and Marder, 2014). Rater error is a major threat to the reliability and validity of classroom observation scores and principal ratings, despite great effort to train raters and calibrate their scoring (Casabianca, Lockwood, and McCaffrey, 2015;Myford, 2012;Whitehurst, Chingos, and Lindquist, 2014). Using Classroom Assessment Scoring System data from the recent Measures of Effective Teaching project, Drew Gitomer and his colleagues found that observers had difficulty agreeing on ratings of certain dimensions of teaching, particularly those for which teacher performance was generally relatively weak (Gitomer et al, 2014).…”