School-to-School Differences in Instructional Practice: New Descriptive Evidence on Opportunity to Learn

Abstract

Background/Context There is continuing debate among social scientists and educators about the role of school-to-school differences in generating educational inequality. Are some students high achieving because they attend School A, while others struggle because they attend School B, as critical discourse on schools argues? Alternatively, is educational inequality driven largely by social forces outside of the school, in the home and neighborhood environment, or by educational processes that are largely common across schools as much social science research argues? Analyses of school achievement, and in particular test score gains from year-to-year, suggest very small between-school differences. Yet, analyses of test score data alone may fail to reveal important school-to-school differences that affect the quality of the classroom experience and a variety of educational outcomes. Purpose/Objective We provide evidence on the following research questions. What is the magnitude of school-to-school variation in instructional practice, as captured by multiple measures? Are some domains of instruction (e.g., behavioral management) more variable between schools than others? To what extent are school-to-school differences in instruction associated with compositional characteristics of students and teachers? Research Design This study relies on the Measures of Effective Teaching Study data, which offer an unprecedented set of observations of teachers’ instruction scored on state-of-the-art observational protocols. To examine the extent of school-to-school variation in instructional practice in elementary and middle schools, we conducted a decomposition of variance analysis using summary scores on multiple measures. We further examine behavioral climate as revealed during instruction separately from overall instructional practice. Next, we examine differences in instruction associated with compositional characteristics of students using multilevel models. Finally, we use an innovative two-stage statistical adjustment strategy to more narrowly identify the possible association between composition and teaching practice due to school-to-school teacher sorting. Findings/Results The basic descriptive results from this study suggest a middle view of school-to-school differences in instruction. We find that substantial school-level variation in instruction exists, with 30% or more of the total variance in instruction lying between schools in these data. Behavioral climate during instruction appears to be particularly salient, and especially in elementary schools. Much of the between school variance we identify, in some cases 40% or more, is readily explained by simple measures of socio-demographic composition, including in particular the racial make-up of schools in the MET districts. Finally, some evidence from a statistical adjustment method suggests that teacher sorting, rather than measurement bias and teacher adaptation, is principally responsible for school-to-school differences in instruction. Conclusions/Recommendations More than an academic debate, basic differences between schools in the quality of the learning environment, along with parental understandings and beliefs about school effects, are potentially important drivers of school and neighborhood sorting and segregation, and even public investment in schooling. Additionally, this question carries continued policy relevance as states adopt and revise teacher and school accountability frameworks that implicitly attribute school-to-school differences to organizational functioning, and seek to carry out instructional improvement efforts in targeted schools. The basic descriptive results from this study suggest school-level differences are not as great as suggested by critical theory and the public discourse, but neither are they as inconsequential as one might infer from some social science research or the literature on value-added differences between schools.

Agnosticism in instructional observation systems

“…That is, where observation is carried out using codes developed a-priori, not that emerge during coding. 1 Another important similarity is that both the agnostic and non-agnostic approaches described here can be used to study and document variation in opportunity to learn (Kelly et al, 2020b). In that way, both can be employed for discovery-oriented research to understand the sources of educational inequality.…”