Should value‐added school effects models include student‐ and school‐level covariates? Evidence from Australian population assessment data

Abstract

There is an enduring issue on whether student-and school-level covariates should be included in value-added school effects models, in addition to prior achievement. Proponents argue that the addition of covariates allows fairer comparisons of schools, whereas opponents argue that it excuses poorly performing schools and obscures policy-relevant school differences. School-level covariates are problematic statistically, but it has been argued that mean school prior achievement should be included in school effects analyses to reduce error. This article reports on school effects analyses of Australia-wide data of approximately 1.5 million students in both primary and secondary schools that took national assessments in five achievement domains between 2013 and 2018. With appropriate controls for prior achievement, school effects are generally small and most often not statistically significant. The addition of student-level covariates: further reduces school effects, since part of the school effects is absorbed by the effects of the covariates, which are unlikely to reflect causal social processes; reduces the proportion of schools with significant school effects; does not improve predictive power; increases the amount of missing data; and further reduces the consistency of school effects between domains and their stability over time. Mean school prior achievement did not improve consistency or stability. Incorporating covariates in school effects analyses opens a Pandora's Box of specification and measurement issues, undermining the legitimacy of school comparisons. It is concluded that researchers and administrators of educational jurisdictions should focus mainly on simpler models based on prior achievement.

Inadequacies in the SES–Achievement model: Evidence from PISA and other studies

“…For Australia, intradomain correlations in the National Assessments-Literacy and Numeracy (NAPLAN) range from 0.6 to 0.9 and are higher for numeracy and spelling than for other domains, and higher in secondary school than in primary school (Marks, 2021). For Germany, the correlation in mathematics test scores (one based on PISA) taken one year apart in 2003 and 2004 was 0.73 (Kriegbaum et al, 2015).…”

A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses and challenges for ‘Progress 8’ raised by COVID‐19

“…For example, if the association between poverty and lower performance chiefly reflects the impact of poorer teaching standards in deprived areas (rather than being driven by factors external to the school environment) then controlling for student disadvantage might remove this variation despite it being an important component to how well a school is serving its student population (Perry, 2016). Marks (2020) also argues that adjusting for student background characteristics may well introduce measurement issues.…”

Students’ Satisfaction towards E-Assessment for Academic Achievement in ESL at Public Schools and Colleges

“…In Pakistan, generally, while in Sindh mainly, English language teaching, learning, and assessment have remained a significant challenge for students and teachers. According to the views shared in [3], to use assessment more, assessment should also be an element that can motivate students to learn [4]. It has been shown that assessment tools should be used to assess or evaluate students' achievement and enhance the quality of language learning and teaching, but this is missing in the Pakistani context and is one of the main issues of assessment.…”