Value-Added Modeling of Teacher Effectiveness: An Exploration of Stability across Models and Contexts

Abstract

Recent policy interest in tying student learning to teacher evaluation has led to growing use of value-added methods for assessing student learning gains linked to individual teachers. VAM analyses rely on complex assumptions about the roles of schools, multiple teachers, student aptitudes and efforts, homes and families in producing measured student learning gains. This article reports on analyses that examine the stability of high school teacher effectiveness rankings across differing conditions. We find that judgments of teacher effectiveness for a given teacher can vary substantially across statistical models, classes taught, Education Policy Analysis Archives Vol. 18 No. 23 2 and years. Furthermore, student characteristics can impact teacher rankings, sometimes dramatically, even when such characteristics have been previously controlled statistically in the value-added model. A teacher who teaches less advantaged students in a given course or year typically receives lower effectiveness ratings than the same teacher teaching more advantaged students in a different course or year. Models that fail to take student demographics into account further disadvantage teachers serving large numbers of low-income, limited English proficient, or lower-tracked students. We examine a number of potential reasons for these findings, and we conclude that caution should be exercised in using student achievement gains and value-added methods to assess teachers' effectiveness, especially when the stakes are high.Keywords: Value-Added Models; Teacher Effectiveness; Accountability and Policy. Modelos de Valor Añadido de la Eficacia DocenteResumen: El reciente interés para relacionar los resultados de los aprendizajes de los estudiantes con las evaluaciones docentes ha provocado un uso creciente de Métodos de Valor Añadido (MVA) para evaluar incrementos en los aprendizajes de los alumnos. Los análisis MVA se basan en hipótesis complejas acerca del papel que desempeñan las escuelas, los profesores, las aptitudes de los estudiantes, sus esfuerzos, sus hogares y familias en la producción de incrementos medibles de aprendizaje estudiantil . Este artículo discute los análisis que examinan la estabilidad de las clasificaciones de eficacia de profesores de escuelas secundarias a través de diferentes condiciones. Encontramos que valoraciones de la efectividad docente de un profesor/a pueden variar substancialmente a través de los distintos modelos estadísticos, las clases enseñadas y los años de experiencia. Además, las características de los estudiantes pueden impactar drásticamente las clasificaciones de los profesores, aún cuando tales características hayan sido previamente controladas estadísticamente en un MVA. Los profesores que enseñan a los estudiantes menos aventajados en un curso o año determinado reciben clasificaciones de eficacia más bajas que cuando un mismo profesor está enseñando a los estudiantes más aventajados en otro curso o año. Los modelos que no toman en cuenta las características demográficas de los es...

Value-added modeling: A review

“…Critics of using value-added in this capacity raise a number of concerns, of which the most prominent are (1) valueadded estimates may be biased (Baker et al, 2010, Paufler andAmrein-Beardsley, 2014;Rothstein, 2009Rothstein, , 2010, and (2) value-added estimates seem too unstable to be used for high-stakes personnel decisions (Baker et al, 2010;Newton et al, 2010). Rothstein (2015) also raises the general point that the labor-supply response to more rigorous teacher evaluations merits careful attention in the design of evaluation policies.…”

Between Scylla and Charybdis: Reflections on and problems associated with the evaluation of teachers in an era of metrification

“…Moreover, this source of variance seems difficult to remove statistically (cf. Newton, Darling-Hammond, Haertel, & Thomas, 2010).…”

Eficácia docente no ensino da matemática

“…Em artigo sobre a estabilidade das classificações dos professores de Ensino Médio (EM) em condições diferentes, Newton et al (2010) descobriram em sua pesquisa nos Estados Unidos que as avaliações de eficácia do ensino de um professor podem variar consideravelmente a partir dos diferentes modelos estatísticos, das aulas ministradas e dos anos de experiência e concluíram que é necessário muito cuidado ao se fazer inferências sobre a eficácia dos professores com base em modelos de valor agregado, especialmente para determinar resultados que podem ter consequências graves (tais como a promoção de um ciclo para outro). Para os autores, em resumo, as características dos estudantes podem modificar drasticamente a classificação dos professores, mesmo que tais características tenham sido controladas estatisticamente nos modelos de valor agregado.…”