When Seeing Is Believing: Generalizability and Decision Studies for Observational Data in Evaluation and Research on Teaching

Abstract

Observations are widely used in research and evaluation to characterize teaching and learning activities. Because conducting observations is typically resource intensive, it is important that inferences from observation data are made confidently. While attention focuses on interrater reliability, the reliability of a single-class measure over the course of a semester receives less attention. We examined the use and limitations of observation for evaluating teaching practices, and how many observations are needed during a typical course to make confident inferences about teaching practices. We conducted two studies based on generalizability theory to calculate reliabilities given class-to-class variation in teaching over a semester. Eleven observations of class periods over the length of a semester were needed to achieve a reliable measure, many more than the one to four class periods typically observed in the literature. Findings suggest practitioners may need to devote more resources than anticipated to achieve reliable measures and comparisons.

[Retracted] Evaluation of College Students’ IPE Environment Based on User Rating Sparsity and Big Data Technology

“…Mobbs et al aimed at the challenges of big data in enhancing the availability of university students' IPE and discussed the countermeasures of big data for enhancing the availability of university students' IPE [ 15 ]. Weston et al studied how to use Web mining technology and combine existing clustering technology to achieve high accuracy classification and clustering of Web text data [ 16 ]. Nurse et al built an instructional resource recommendation system based on online education, and designed and implemented each module of the system; describe the development platform and tools, and test the actual development effect [ 17 ].…”

Co-teaching in Undergraduate STEM Education: A Lever for Pedagogical Change toward Evidence-Based Teaching?

“…, 2018 ), but two different research groups have since demonstrated that four observations are often inadequate to reliably document instructional change across time ( Sbeglia et al. , 2021 ; Weston et al. , 2021 ).…”

Comparing perceived and observed instructional practices and their predictive power for student mathematics achievement: An analysis of Shanghai data from OECD global teaching inSights

“…There is ongoing debate about the extent to which observer ratings can be used to judge teachers’ instructional quality. Some researchers argue that observer ratings should be used as one piece of evidence in a broader evaluation process (e.g., Lei et al, 2018; Van der Lans, 2018), while others question the validity and fairness of using such observer ratings (e.g., Praetorius et al, 2014; Weston et al, 2021). On the one hand, some prior studies have found that observer ratings of instructional quality are consistent across different observers and occasions, and that they correlate with other measures of instructional quality, such as student ratings and student learning outcomes (Hill et al, 2012; Van der Lans, 2018).…”