Making Inferences About Teacher Observation Scores Over Time

Abstract

Observation protocol scores are commonly used as status measures to support inferences about teacher practices. When multiple observations are collected for the same teacher over the course of a year, some portion of a teacher’s score on each occasion may be attributable to the rater, lesson, and the time of year of the observation. All three of these are facets that can threaten the generalizability of teacher scores, but the role of time is easiest to overlook. A generalizability theory framework is used in this study to illustrate the concept of a hidden facet of measurement. When there are many temporally spaced observation occasions, it may be possible to support inferences about the growth in teaching practices over time as an alternative (or complement) to making inferences about status at a single point in time. This study uses longitudinal observation scores from the Measures of Effective Teaching project to estimate the reliability of teacher-level growth parameters for designs that vary in the number and spacing of observation occasions over a 2-year span. On the basis of a subsample of teachers scored using the Danielson Framework for Teaching, we show that at least eight observations over 2 years are needed before it would be possible to make distinctions in growth with a reliability coefficients of .39.

When Seeing Is Believing: Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching

“…While nominally class-to-class variation is to be expected, here it is considered error given that we are attempting to generalize what we see in a few class sessions to a whole course term. As noted, this is a common practice in other Gstudies that examine the number of occasions classes have been observed (Briggs & Alzen, 2019;Hill et al, 2012). Some researchers also call these differentiation and instrumentation sources of variance corresponding to true and error variance (Cardinet et al, 2011).…”

Generalizability theory.

“…Since its introduction in Cronbach et al (1972), G theory has emerged as one of the more frequently used modern test theories. Applications of G theory have extended into many scientific fields, including behavioral and social sciences (e.g., see Arterberry et al, 2014;Bergeron et al, 2008;Brennan & Liao, 2020;Briggs & Alzen, 2019;Christ et al, 2010;Clauser et al, 2020;Halvorsen et al, 2006;Lakes & Hoyt, 2009;Lee & Lewis, 2008;Lei et al, 2007;Shavelson et al, 1993;Vispoel et al, 2018;Wasserman et al, 2008). In addition, G theory has been linked to other techniques, such as structural equation modeling (see Ark, 2015, andVispoel et al, 2018).…”

Comparing perceived and observed instructional practices and their predictive power for student mathematics achievement: An analysis of Shanghai data from OECD global teaching inSights

“…First, observer ratings are inherently subjective as observers may bring their own biases, beliefs, interpretation of teaching behaviors to the rating process, causing potential variability in ratings and negatively impact the reliability and validity of such ratings (Praetorius et al, 2014). Because of this, observers need to be rigorously trained on the observation tool or rubric, and then subsequently calibrated to ensure that they are applying the tool or rubric consistently from the beginning to the end (Briggs & Alzen, 2019). Additionally, observers are typically only able to observe a limited portion of a teacher's practice and may not have a complete understanding of the context in which teaching is occurring; such limited scope may also negatively impact the validity and reliability of the ratings (Weston et al, 2021).…”

“…Despite these limitations, observer-rated instructional quality has remained a valuable tool. Obviously, its validity and reliability largely rely on that of the observation tool or rubric (Reddy et al, 2019), whether observers are rigorously trained, and rating calibration is considered (Briggs & Alzen, 2019). Since data collected from observer ratings can possibly provide researchers with rich information about the actual teaching and learning process (Grossman et al, 2014; Hill et al, 2011; Pianta & Hamre, 2009), particularly when video recording of teachers’ lessons are obtained and then analyzed by trained raters, it has gained more popularity in recent decades with the affordance of modern video technology as evidenced by the Video Study of Trends in International Mathematics and Science Study (TIMSS) in the 1990s (Stigler & Hiebert, 2009) and then the Organization for Economic Co-Operation and Development (OECD)'s large scale Global Teaching Insights (GTI) study in 2018 (OECD, 2020a).…”