Measuring Teaching Quality of Secondary Mathematics and Science Residents: A Classroom Observation Framework

Abstract

This study assesses the reliability of two observation rubrics, one in math and the other in science, and documents how the rubric data were used to inform a teacher education program. Classroom observations are typically considered essential for assessing teaching practice, yet many popular observation frameworks, while comprehensive in aim, do not appropriately capture key features of teaching valued by teacher education programs. Many of these tools do not attend to issues of equity, humanizing pedagogy, and thus, social justice. We report on the development of two observation rubrics—secondary math and science—that embody the aims and values of our teacher education program, specifically, equity and humanizing pedagogy, and the results of our examination of the reliability of ratings of teaching practice generated using these rubrics. We discuss the various sources of measurement error and the implications for further developing and using the observation rubric in our program.

What Data and Measures Should Inform Teacher Preparation? Reclaiming Accountability

“…For example, some view what they call the "dominant accountability paradigm" (Cochran-Smith et al, 2018) as reflecting market ideology and neoliberalism in education, privileging subject matter and often disconnected from the experiences and needs of students and teachers from nondominant communities (Lipman, 2011;Richmond, Bartell, & Dunn, 2016). Others call for building on what they see as a rich body of research that identifies knowledge and skills aimed at promoting democratic and socially just education and commitments in practice (e.g., Cochran-Smith et al, 2009;Crowley & Apple, 2009;Kumashiro, 2015;McDonald, 2005;Zeichner, Payne, & Brayko, 2014 This poses a dilemma for teacher education, which we have described in earlier editorials (Richmond et al, 2016): As we necessarily work within a system that warrants accountability, how might we "reclaim" accountability (Cochran-Smith et al, 2018) to support teacher candidates' preparation for the realities of today's schools and for their effective participation in "critical democracy" (Carter Andrews, Richmond, & Floden, 2018)? In this editorial, we call for casting a broader net that is both (a) more comprehensive with respect to the knowledge, skills, and practices evaluated and (b) more inclusive of the types of measures we use to obtain data.…”

The quality and qualities of classroom observation systems

“…Many authors point to issues that need to be considered when selecting and/or using a COS (e.g., Bell et al, 2012;Cohen & Goldhaber, 2016;Hill, Charalambous, Blazer, et al, 2012;Nava et al, 2018;Sandilos, 2012;van der Lans, van de Grift, van Veen, & Fokkens-Bruinsma, 2016). Generating valid and reliable scores by means of a COS is not self-evident.…”

“…A manual can also include guidelines concerning the recommended observation period (e.g., 30 minutes), when to score the observation form (e.g., at the end of the lesson), and how to act during the observation (e.g., stay at the periphery of the action), since this is likely to affect the scores provided by raters. Considering that multiple studies have shown that the reliability of the scores improves if the number of raters increases Nava et al, 2018), manuals can also include guidelines concerning the number of raters per teacher.…”

“…If the observed scores are generalized to a broader domain than what was observed (for example, teaching quality over a school year), which is often the case, then sampling specifications become important as the observation(s) should be representative of the context to which the score is generalized. Important specifications are the required number of observations per teacher Nava et al, 2018), the duration of those observations (Hill, Charalambous, Blazar et al, 2012), the types of lessons (Grossman et al, 2014), and the timing of the observations during the day, week, and year (Casabianca et al, 2015;Sandilos, 2012).…”

“…Generating valid and reliable scores when using a COS is not guaranteed. Many authors (e.g., Bell et al, 2012;Cohen and Goldhaber, 2016;Hill, Charalambous, Blazer, et al, 2012;Nava et al, 2018;Sandilos, 2012;van der Lans, van de Grift, van Veen, & Fokkens-Bruinsma, 2016) point to issues that need to be taken into account when developing and/or using a COS in order to generate valid and reliable scores. For example, issues regarding the number of raters and observations required for obtaining reliable scores.…”