Ecologies of education quality

Abstract

Accountability in education has prompted policy makers and practitioners to focus on data use for instructional and organizational decision-making. The popular media have seized on Value Added (VA) measures as a key type of data use for reforming U.S. schools. Among education researchers, however, there are both critics and proponents. We examined data use by the district leaders and staff members of 12 schools in a large urban district, with attention to the role VA metrics play in their decisions and their conceptions of themselves. VA is only one of many types of data that can be used to portray quality. While there was a soft relationship between VA and classroom quality measured by CLASS, we found that understanding the schools' contexts, particularly the use of resources and the coherence of actions to improve student achievement, greatly enhanced the power of our descriptions. As a result, we suggest that policies promoting multidimensional approaches to quality will better capture the complexity of education. Keywords: value added, quality, ecological framework, data use, reform Resumen: programas de responsabilidad educativa (accountability) han impulsado el "uso de información" (data use) entre responsables de implementar políticas y educadores profesionales tanto para decidir sobre cuestiones de instrucción como en decisiones sobre organización. Los medios de comunicación se han enfocado en los modelos de Valor Añadido (MVA) como un tipo clave de "uso de información" para reformar las escuelas estadounidenses. Entre los investigadores en educación, sin embargo, hay tanto críticos y como proponentes de MVAs. Examinamos el "uso de información" por directores, profesionales en 12 escuelas en un distrito urbano de gran tamaño, especialmente el papel que las medidas de MVAs tuvieron en sus decisiones y sus concepciones sobre sí mismos. MVAs es sólo uno de los muchos tipos de "usos de información" que pueden ser utilizados para indiicar calidad. Si bien hay una relación moderada entre MVA y calidad de la enseñanza medidas por CLASS, encontramos que la comprensión de los contextos escolares, especialmente el uso de los recursos y la coherencia de las acciones para mejorar el logro de los estudiantes, aumentó en gran medida el poder de nuestras descripciones. Sugerimos que las políticas que promueven enfoques multidimensionales de calidad captan mejor la complejidad de los sistemas educativos. Palabras clave: modelos de valor agregado; calidad; marco ecológico; uso de información; reforma. Ecologias de qualidade educativa.Resumo: Os programas de A responsabilidade educacionais (accountability) estão empurrando o "uso da informação" (uso de dados) aos responsáveis pela implementação de políticas e educadores nos processos de toma de decisões sobre questões educacionais e organizacionais. Os meios de comunicação têm-se centrado em modelos de valor agregado (MVA) como a questão chave no "uso de informação" para reformar as escolas americanas. Entre os pesquisadores de educação, no entanto, existem tant...

Putting teacher evaluation systems on the map: An overview of states' teacher evaluation systems post–Every Student Succeeds Act

“…These controversial views led to court challenges to states' VAM-based teacher evaluation systems (i.e., in Florida, Louisiana, Nevada, New Mexico, New York, Tennessee, and Texas; see Education Week, 2015). 3 Plaintiffs argued the following main points of criticism regarding VAM models within teacher evaluations systems including that VAMs can be: (1) unreliable, whereby current research suggests that teachers classified as "effective" one year will have a 25%-59% chance of being classified as "ineffective" the next year, or vice versa, with other permutations possible (Chiang, McCullough, Lipscomb, & Gill, 2016;Martinez, Schweig, & Goldschmidt, 2016;Schochet & Chiang, 2013;Shaw & Bovaird, 2011;Yeh, 2013); (2) invalid, whereby very limited research evidence supports the claim that VAMs can be used to draw accurate inferences about the extents to which different teachers cause changes (i.e., add value) in a collective groups of students' test performance over time (see, for example, Amrein-Beardsley, 2008;Braun, 2005Braun, , 2015Hill, Kapitula, & Umland, 2011); (3) biased, whereby current research suggests that, almost regardless of the sophistication of the statistical controls used to block bias, VAM-based estimates sometimes present biased results, especially when relatively homogeneous sets of students (i.e., ELLs, gifted and special education students, free-or-reduced lunch eligible students) are non-randomly concentrated in schools and teachers' classrooms (Baker et al 2010;Capitol Hill Briefing, 2011;Collins, 2014;Green, Baker, & Oluwole, 2012;Kappler Hewitt, 2015;Koedel, Mihaly, & Rockoff, 2015;McCaffrey, Lockwood, Koretz, Louis, & Hamilton, 2004;Newton, Darling-Hammond, Haertel, & Thomas, 2010;Rothstein & Mathis, 2013); (4) not transparent, with the main issue being that VAMbased estimates do not often make sense to those at the receiving ends of the estimates (e.g., teachers and principals) and, subsequently, these same groups are reportedly quite-to-very unlikely to use VAM-based output for formative purposes (see, for example, Eckert & Dabrowski, 2010;Gabriel & Lester, 2013;Goldring et al, 2015;Graue, Delaney, & Karch, 2013); and (5) unfair, with the fundamental issue being that states and districts can only produce VAM-based estimates for approximately 30-40% of all teachers, leaving the other 60-7...…”

“Houston, We Have a Lawsuit”: A Cautionary Tale for the Implementation of Value-Added Models for High-Stakes Employment Decisions

“…A complete overview of the technical criticisms surrounding VAMs is beyond the scope of this review and can be found elsewhere (see, e.g., American Educational Research Association Council, 2015; American Statistical Association, 2014; Amrein-Beardsley, 2008, 2014; Baker et al, 2010; Harris & Herrington, 2015), as can literature reflecting the potential benefits of using VAMs for teacher evaluation purposes (American Educational Research Association Council, 2015; American Statistical Association, 2014; Bill & Melinda Gates Foundation, 2013; Chetty et al, 2014a, 2014b; Goldhaber, 2015; Harris & Herrington, 2015), but a brief sketch of the concerns regarding VAMs is warranted because the concerns reviewed do appear in the Houston decision. These concerns relate primarily to the following terms, as defined in Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 2014): reliability (see, e.g., Martínez, Schweig, & Goldschmidt, 2016; Schochet & Chiang, 2013; Yeh, 2013), validity (see, e.g., Hill et al, 2011; Kane et al, 2012; Martínez et al, 2016); bias (see, e.g., Koedel, Mihaly, & Rockoff, 2015; Newton, Darling-Hammond, Haertel, & Thomas, 2010; Rothstein, 2009), transparency (see, e.g., Eckert & Dabrowski, 2010; Goldring et al, 2015; Graue, Delaney, & Karch, 2013), and fairness (see, e.g., Gabriel & Lester, 2013; Harris, 2011; Jiang et al, 2015). Although the court did not overturn the district’s use of VAMs on the basis of statistical concerns as a matter of substantive due process, the court’s analysis on procedural due process notes these concerns, especially as related to transparency and fairness.…”

Value-Added Models for Teacher Evaluation and Accountability: Commonsense Assumptions

“…D. Baker et al, 2013; E. L. Baker et al, 2010; Briggs & Domingue, 2011; Corcoran, 2010; Eckert & Dabrowski, 2010; Gabriel & Lester, 2013; Graue et al, 2013; Hill et al, 2011; Newton et al, 2010; Papay, 2011; Rothstein, 2009, 2010, 2014; Schochet & Chiang, 2012). Accordingly, VAMs continue to be the source of concern surrounding discussions of educational reform.…”

“…D. Baker, Oluwole, & Green, 2013; E. L. Baker et al, 2010; Briggs & Domingue, 2011; Corcoran, 2010; Eckert & Dabrowski, 2010; Gabriel & Lester, 2013; Graue, Delaney, & Karch, 2013; Hill, Kapitula, & Umlan, 2011; Newton, Darling-Hammond, Haertel, & Thomas, 2010; Papay, 2011; Rothstein, 2009, 2010, 2014). Given the still current momentum of VAM adoption and use, then, it is reasonable to posit that economists have more or less taken the lead in influencing now state-level policies in this area, as well as international initiatives and policies in this area (Araujo, Carneiro, Cruz-Aguayo, & Schady, 2016; Sørensen, 2016), which is not surprising in light of their rising influence in the public policy arena at large (Fourcade, Ollion, & Algan, 2015; Lazear, 1999, 2001)…”