Educational effectiveness: the development of the discipline, the critiques, the defence, and the present debate

Abstract

Educational effectiveness research (EER) has made a significant contribution to our understanding of the characteristics and processes associated with more and less effective schools in a diverse range of contexts. However, this remains a contested field of inquiry and has been subjected to significant critique. This paper examines the origins and development of EER and summarises the key critiques and defences of the field during the past 30 years. It then moves on to examine the recent critique of the field by Stephen Gorard in the UK and responds by highlighting statistical errors and simplistic claims made by Gorard about the field's involvement with the development of national value-added systems and interaction with policy-making in his recent papers Q2 .

Can schools reduce bullying? The relationship between school characteristics and the prevalence of bullying behaviours

“…Educational effectiveness research has frequently been criticized for a lack of attention to non‐cognitive outcomes, defined as those outcomes that are not related to curriculum subjects such as attainment in Maths (which are known as cognitive outcomes), but relate to broader aspects such as social and emotional development. While Reynolds, Chapman, Kelly, Muijs, and Sammons () have argued this critique is only partially justified, recent overviews have shown that the majority of studies in the field still relate to cognitive outcomes by a ratio of about 4 to 1 (Chapman, Muijs, Reynolds, Sammons, & Teddlie, ). Bullying, as a persistent and highly harmful phenomenon in schools, is therefore both a majorly important factor in its own right and an outcome that presents us with an important test of the existence of a relationship between school and classroom processes and non‐cognitive outcomes (Kyriakides et al ., ).…”

‘Phantom’ compositional effects in English school value-added measures: the consequences of random baseline measurement error

“…This negative position on error is contested, and led to a series of exchanges between Gorard and prominent educational effectiveness researchers (Gorard 2010;Muijs et al 2011;Gorard 2011a;Gorard 2011b;Reynolds et al 2012). Defenders of value-added, over the course of these exchanges with Gorard, propounded their view that value-added school effects are sufficiently large, stable and consistent to be of value and that the error tends to be a much smaller component of the variance that, moreover, tends to be random (Reynolds et al 2012) and amenable to technical solutions (Muijs et al 2011). These exchanges raised several issues around bias and error within value-added estimation.…”

“…If this is the case, the argument goes, pupil-level errors will generally cancel out and error is 'unlikely to be systematically different in different schools'. Reynolds et al (2012) go on to cite several studies that have examined the influence of random measurement error within multi-level models (Woodhouse et al 1996;Goldstein, Kounali, and Robinson 2008;Ferrão and Goldstein 2009). These studies give a consistent picture of the effects of random measurement error in multi-level value-added models, suggesting that when models are adjusted for a lower reliability (see Ferrão and Goldstein 2009, 954) of the measure of prior attainment (ranging from 1 down to 0.6 across studies) this leads to a) an increase in the prior attainment coefficient values, b) a decrease in pupil-level variance, c) an increase in the intra-school correlation, an indicator of the importance of the school effect, and d) school-level variance being largely unaffected.…”

Getting lost in translation? An analysis of the international engagement of practitioners and policy-makers with the educational effectiveness research base

“…Over the years, these critiques have been robustly countered (e.g. Reynolds and Teddlie 2001) and SER researchers continue to challenge general misunderstandings pertaining to statistics and the field's relationship with policy-makers (Reynolds et al 2012). Notwithstanding these debates the EER knowledge base has provided:…”