Validating Arguments for Observational Instruments: Attending to Multiple Sources of Variation

Abstract

Measurement scholars have recently constructed validity arguments in support of a variety of educational assessments, including classroom observation instruments. In this article, we note that users must examine the robustness of validity arguments to variation in the implementation of these instruments. We illustrate how such an analysis might be used to assess a validity argument constructed for the Mathematical Quality of Instruction instrument, focusing in particular on the effects of varying the rater pool, subject matter content, observation procedure, and district context. Variation in the subject matter content of lessons did not affect rater agreement with master scores, but the evaluation of other portions of the validity argument varied according to the composition of the rater pool, observation procedure, and district context. These results demonstrate the need for conducting such analyses, especially for classroom observation instruments that are subject to multiple sources of variation.

Student Teaching Feedback and Evaluation: Results From a Seven-State Survey

“…Efforts invested in this pursuit are likely to benefit both the program and the students being evaluated as research studies have demonstrated that when validated tools are used for formative teacher evaluation and tailored feedback, they can successfully improve teaching quality (Allen, Pianta, Gregory, Mikami, & Lun, 2011;McCollum, Hemmeter, & Hsieh, 2011). However, if these tools are implemented poorly or inconsistently, the promised returns diminish (Hill et al, 2012).…”

Relationships between Observations of Elementary Mathematics Instruction and Student Achievement: Exploring Variability across Districts

“…Teacher ambitious instruction scores were generated from videotaped lessons of instruction captured over the course of 2 years. Teachers averaged 5.23 videotaped lessons (mode p 6), allowing for sufficient levels of predictive reliability (Hill, Charalambous, Blazar, et al 2012). 1 Raters had a background in mathematics or mathematics education, passed a certification exam, and completed ongoing calibrations.…”

Development of a Student Self-Reported Instrument to Assess Course Reform

“…The CEQ was catalyzed by the instruments named; however, the CEQ was specifically designed to address the temporal and fiscal limitations of observation research as well as the limitations of instructor self-reporting methods (Hill et al, 2012). The process of creating the CEQ involved an adaptation and transformation of constructs present on the cited protocols, such that selected items were translated into classroom practices that could be measured by student self-report as well as through observational methods.…”

“…A two-way mixed model was necessary because the study design called for data collection from a fixed group of observers and a random sample of students (Haber, Barnhart, Song, & Gruden, 2005;Hill et al, 2012;Shrout & Fleiss, 1979).…”

“…It is difficult, if not impossible, to create a standardized measurement tool that is applicable to all reformed teaching and learning practices (Arum & Roksa, 2010). Too, the resource-intensive costs of evaluating teaching through direct observation methods steer many programs toward other data-gathering methods that are less directly connected to the learning environment (Hill et al, 2012), often leading programs to measure the efficacy of reform through faculty self-reports of (reformed) teaching practices (Kuh, 2003).…”