Bringing the Theory and Measurement of Teaching into Alignment

Abstract

There is a growing use of standardized observation systems to directly measure teaching quality in classrooms. These systems are based on rich, theoretical understandings of teaching quality that lead to carefully operationalized rubrics that decompose teaching quality into a number of distinct dimensions. In this paper, we argue that many of the ways observation systems are applied and analyzed do not fully conform to the theoretical understandings of teaching quality upon which the observation system is based. We further argue for the need to align theoretical understandings of teaching quality with analytical approaches to measure teaching quality. We discuss four key assumptions that undergird many views of teaching quality and highlight how common analytical approaches violate these assumptions, proposing alternative analytical approaches that would better conform to the theoretical understandings of teaching quality. We end with a discussion of the importance of carefully aligning theoretical understandings with measurement approaches.

Agnosticism in instructional observation systems

“…Subsequently, in my view, a great deal of research then focuses narrowly on traditional measurement properties (i.e., the most common expressions of reliability and validity), without much consideration of the basic features outlined in Table 1 and how they affect the results. However, this focus is beginning to change, as researchers interrogate how these features might function (Hennessy et al, 2020;White et al, 2021). The present discussion seeks to add to that changing focus by highlighting agnosticism in particular.…”

“…Before proceeding with a continued discussion of agnosticism, I offer a brief discussion of other fundamental features of classroom observation protocols, summarized in Jarvis et al (2003) provide an overview of this distinction. White et al (2021) discuss this at length with regard to teacher observation. See also Kelly et al (2020).…”

“…In various ways, if they are flawed measures, this may negatively impact use. There is a literature investigating the measurement properties of global observation protocols (Bell et al, 2014;Campbell & Ronfeldt, 2018;Cohen & Goldhaber, 2016;Gitomer et al, 2014;Humphry & Heldsinger, 2014;Liu et al, 2019;McCaffrey et al, 2015;White, 2018;White et al, 2021), and some of that literature offers very sharp critique (Kelly et al, 2020). To describe just one concern, scores on the various sub-domains of global protocols, ostensibly important to providing teachers useful, domain-specific feedback, are not anywhere near as separable in practice as the structure of the protocols would make them out to be (Aucejo et al, 2022;Humphry & Heldsinger, 2014;Liu et al, 2019;McCaffrey et al, 2015).…”