Stability over time of different methods of estimating school performance

Dumay, Xavier; Coe, Rob; Anumendem, Dickson

doi:10.1080/09243453.2012.759599

Cited by 26 publications

(37 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Guldemond and Bosker (2009) found intra-class correlations of 0.3 to 0.5 in growth models. Dumay, Coe, and Anumendem (2014) found that, on average, across cohorts 74% of the total slope variance was accounted for by the school-level slope variance, whereas residual gain scores analysis showed a proportion attributable to schools of just 16% on average, across cohorts. Anumendem, De Fraine, Onghena, and Van Damme (2017) showed considerable discrepancies between intra-class correlations expressed as adjusted performance status and growth; according to one of their models, this discrepancy is as high as 0.18 for performance versus 0.66 for the growth model.…”

Section: School Effects In Terms Of Student Progressmentioning

confidence: 88%

“…Such interpretations could be seen as useful in practical applications such as school self-evaluations and external school evaluation. The relatively low stability of the growth curves coefficients over time, as compared to estimates based on adjusted performance status (Dumay et al, 2014;Timmermans & Van der Werf, this issue), would make this method more problematic for such practical applications. But if growth curves analysis would indeed prove the most internally valid way of assessing net effects of schooling, it might become the preferred method in educational effectiveness research.…”

Section: Discussionmentioning

confidence: 99%

“…Conceptually, the coefficients yielded by this method are interesting as they can be thought of as efficiently isolating the part of the between-school variables from exogenous variables (Dumay et al, 2014). Yet, the slope ratios on which the school effects are based may be lacking a sound statistical rationale.…”

Section: Discussionmentioning

confidence: 99%

“…Palardy (2008) reported more modest discrepancies in the intra-class correlations for performance status and growth (.20 versus .23 in the full sample, 13 versus .25 and .07 versus .14 among low-and high-scoring schools, respectively). School effects obtained from growth models are only weakly associated with students' socioeconomic status (Dumay et al, 2014;Guldemond & Bosker, 2009). Moreover, the school effects obtained from growth models were less stable than school effects obtained by adjusting for initial performance (Dumay et al, 2014).…”

Section: School Effects In Terms Of Student Progressmentioning

confidence: 99%

“…School effects obtained from growth models are only weakly associated with students' socioeconomic status (Dumay et al, 2014;Guldemond & Bosker, 2009). Moreover, the school effects obtained from growth models were less stable than school effects obtained by adjusting for initial performance (Dumay et al, 2014). Large school effects were also found in the growth models analysed by Timmermans and Van der Werf in this special issue.…”

Section: School Effects In Terms Of Student Progressmentioning

confidence: 99%

See 4 more Smart Citations

The perspective of “limited malleability” in educational effectiveness: treatment effects in schooling

Scheerens

2017

Educational Research and Evaluation

View full text Add to dashboard Cite

In this article, several ways to adjust gross school effects are discussed to set the stage for estimating treatment effects in schooling. Although it is quite hazardous to hypothesize realistic benchmarks for results from meta-analyses, because of the dependency of effect sizes on subject matter area, grade level, and study characteristics, a treatment effect of d = .20 can be considered as educationally meaningful. Commonly addressed effectiveness-enhancing school factors show much inconsistency across meta-analyses and frequently small effects (in the order of d = .10). An analysis of the conceptual and methodological foundations of defining and measuring treatment effects in schooling indicates that flaws in the measurement of treatments could be an additional cause of small effect sizes in educational effectiveness research. Results of this review are analysed from the perspective of "limited malleability" in educational effectiveness. KEYWORDSEducational effectiveness; effect sizes; absolute and relative effects of schooling

show abstract

Section: School Effects In Terms Of Student Progressmentioning

confidence: 88%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: School Effects In Terms Of Student Progressmentioning

confidence: 99%

Section: School Effects In Terms Of Student Progressmentioning

confidence: 99%

See 3 more Smart Citations

The perspective of “limited malleability” in educational effectiveness: treatment effects in schooling

Scheerens

2017

Educational Research and Evaluation

View full text Add to dashboard Cite

show abstract

English Value‐Added Measures: Examining the Limitations of School Performance Measurement

Perry

2016

British Educational Res J

View full text Add to dashboard Cite

Value-added 'Progress' measures are to be introduced for all English schools in 2016 as 'headline' measures of school performance. This move comes despite research highlighting high levels of instability in value-added measures and concerns about the omission of contextual variables in the planned measure. This article studies the impact of disregarding contextual factors, the stability of school scores across time and the consistency of value-added performance for different cohorts within schools at a given point in time. The first two analyses replicate and extend previous studies using current data, confirming concerns about intake biases and showing that both secondary and primary level value-added measures exhibit worrying levels of instability. The third analysis goes further by examining whether instability across time is likely to stem from differences between cohorts and whether measures based on a single cohort reflect school performance more generally. Combined, these analyses suggest a general problem of imprecision within value-added estimates and that current policy use of value-added is unjustified. Published school performance measures are likely to be profoundly misleading, in particular for those unfamiliar with the level of uncertainty in the estimates. The article closes by considering whether value-added measures could and should be used by policy-makers as measures of school performance.

show abstract

Should value‐added school effects models include student‐ and school‐level covariates? Evidence from Australian population assessment data

Marks

2020

British Educational Res J

View full text Add to dashboard Cite

There is an enduring issue on whether student-and school-level covariates should be included in value-added school effects models, in addition to prior achievement. Proponents argue that the addition of covariates allows fairer comparisons of schools, whereas opponents argue that it excuses poorly performing schools and obscures policy-relevant school differences. School-level covariates are problematic statistically, but it has been argued that mean school prior achievement should be included in school effects analyses to reduce error. This article reports on school effects analyses of Australia-wide data of approximately 1.5 million students in both primary and secondary schools that took national assessments in five achievement domains between 2013 and 2018. With appropriate controls for prior achievement, school effects are generally small and most often not statistically significant. The addition of student-level covariates: further reduces school effects, since part of the school effects is absorbed by the effects of the covariates, which are unlikely to reflect causal social processes; reduces the proportion of schools with significant school effects; does not improve predictive power; increases the amount of missing data; and further reduces the consistency of school effects between domains and their stability over time. Mean school prior achievement did not improve consistency or stability. Incorporating covariates in school effects analyses opens a Pandora's Box of specification and measurement issues, undermining the legitimacy of school comparisons. It is concluded that researchers and administrators of educational jurisdictions should focus mainly on simpler models based on prior achievement.

show abstract

Stability over time of different methods of estimating school performance

Cited by 26 publications

References 24 publications

The perspective of “limited malleability” in educational effectiveness: treatment effects in schooling

The perspective of “limited malleability” in educational effectiveness: treatment effects in schooling

English Value‐Added Measures: Examining the Limitations of School Performance Measurement

Should value‐added school effects models include student‐ and school‐level covariates? Evidence from Australian population assessment data

Contact Info

Product

Resources

About