When is a Compositional Effect not a Compositional Effect?

Hutchison, Dougal

doi:10.1007/s11135-007-9094-2

Cited by 18 publications

(14 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The effects of variables aggregated at the school level have been described as phantom effects (Harker & Tymms, 2004;Pokropek, 2014;Televantou et al, 2015;Perry, 2019). Hutchison (2007) shows mathematically that spurious effects of aggregated variables occur even in properly specified models and remain as sample sizes approach infinity.…”

Section: Correlation Of School Effects Across Statistical Modelsmentioning

confidence: 99%

Should value‐added school effects models include student‐ and school‐level covariates? Evidence from Australian population assessment data

Marks

2020

British Educational Res J

View full text Add to dashboard Cite

There is an enduring issue on whether student-and school-level covariates should be included in value-added school effects models, in addition to prior achievement. Proponents argue that the addition of covariates allows fairer comparisons of schools, whereas opponents argue that it excuses poorly performing schools and obscures policy-relevant school differences. School-level covariates are problematic statistically, but it has been argued that mean school prior achievement should be included in school effects analyses to reduce error. This article reports on school effects analyses of Australia-wide data of approximately 1.5 million students in both primary and secondary schools that took national assessments in five achievement domains between 2013 and 2018. With appropriate controls for prior achievement, school effects are generally small and most often not statistically significant. The addition of student-level covariates: further reduces school effects, since part of the school effects is absorbed by the effects of the covariates, which are unlikely to reflect causal social processes; reduces the proportion of schools with significant school effects; does not improve predictive power; increases the amount of missing data; and further reduces the consistency of school effects between domains and their stability over time. Mean school prior achievement did not improve consistency or stability. Incorporating covariates in school effects analyses opens a Pandora's Box of specification and measurement issues, undermining the legitimacy of school comparisons. It is concluded that researchers and administrators of educational jurisdictions should focus mainly on simpler models based on prior achievement.

show abstract

Section: Correlation Of School Effects Across Statistical Modelsmentioning

confidence: 99%

Should value‐added school effects models include student‐ and school‐level covariates? Evidence from Australian population assessment data

Marks

2020

British Educational Res J

View full text Add to dashboard Cite

show abstract

“…Although there is strong empirical evidence and a large number of studies to support both the BFLPE (see Marsh & Seaton, 2015 for an overview) and the REM (Marsh & Craven, 2006), results regarding the peer spillover effect, although widely accepted, are inconsistent (Hattie, 2002; Hutchison, 2007; Nash, 2003; Televantou et al, 2015). Moreover, methodological advances show that the presence of such positive compositional effects of achievement might be the result of unreliability at the individual level for which has not been accounted for.…”

mentioning

confidence: 99%

Effects of school-average achievement on individual self-concept and achievement: Unmasking phantom effects masquerading as true compositional effects.

Dicke¹,

Marsh²,

Parker³

et al. 2018

Journal of Educational Psychology

View full text Add to dashboard Cite

School-average achievement is often reported to have positive effects on individual achievement (peer spillover effect). However, it is well established that school-average achievement has negative effects on academic self-concept (big-fish-little-pond effect [BFLPE]) and that academic self-concept and achievement are positively correlated and mutually reinforcing (reciprocal effects model). We resolve this theoretical paradox based on a large, longitudinal sample (N ϭ 14,985 U.S. children) and improved methodology. More appropriate multilevel modeling that controls for phantom effects (due to measurement error and preexisting differences) makes the BFLPE even more negative, but turns the peer spillover effect from positive to slightly below zero. Thus, attending a high-achieving school has negative effects on academic self-concept and a nonpositive effect on achievement. The results question previous studies and meta-analyses showing a positive peer spillover effect that do not control for phantom effects, along with previous policy and school selection decisions based on this research. Educational Impact and Implications StatementCounter to a widely held belief that being in a high-achieving school has a positive effect on student's achievement (peer spillover effect), the present findings suggest that this effect is actually slightly negative. When using stronger, more appropriate statistical methodology, the apparent peer spillover effect disappeared, suggesting that positive effects are a phantom. Furthermore, the negative effect of school-average achievement on academic self-concept (big-fish-little-pond effect) turned out to be even more negative when using more appropriate methodology. Thus, the findings indicate that attending a high-achieving school has a negative effect on self-concept and no positive effect on achievement. These results call into question prior research that did not control for phantom effects, and challenge policy and practice decisions that promote selective schooling.

show abstract

“…As Gorard (2006a) notes, the difficulties of estimating school effects using noisy data are already sizable and these are compounded when studying compositional effects which are a fraction of the overall school effect. There are now several methodological studies that have shown that spurious (or 'phantom') compositional effects can arise from pupil-level measurement error (Harker and Tymms 2004;Hutchison 2007;Pokropek 2014;Televantou et al 2015) or the failure to take pupil-level control variables such as SES into account (Nash 2003).…”

Section: Phantom Compositional Effectsmentioning

confidence: 99%

‘Phantom’ compositional effects in English school value-added measures: the consequences of random baseline measurement error

Perry

2018

Research Papers in Education

View full text Add to dashboard Cite

Where a licence is displayed above, please note the terms and conditions of the licence govern your use of this document. When citing, please reference the published version. Take down policy While the University of Birmingham exercises care and attention in making items available there are rare occasions when an item has been uploaded in error or has been deemed to be commercially or otherwise sensitive.

show abstract

When is a Compositional Effect not a Compositional Effect?

Abstract: The question of compositional effects (that is, the effect of collective properties

Cited by 18 publications

References 14 publications

Should value‐added school effects models include student‐ and school‐level covariates? Evidence from Australian population assessment data

Should value‐added school effects models include student‐ and school‐level covariates? Evidence from Australian population assessment data

Effects of school-average achievement on individual self-concept and achievement: Unmasking phantom effects masquerading as true compositional effects.

‘Phantom’ compositional effects in English school value-added measures: the consequences of random baseline measurement error

Contact Info

Product

Resources

About