International Large-Scale Assessments: Thermometers, Whips or Useful Policy Tools?

Abstract

Over the past three decades, the number of developing countries participating in International Large-Scale Assessments (ILSAs) has increased dramatically, while developing countries' usage of data from ILSAs for informing educational policy has not been fully realised. In this paper the authors argue that for ILSAs to be useful policy tools, alignment must occur between ILSAs' measurement quality mission and their capacity development mission. They review the use of ILSAs as both 'whips' and 'thermometers' and discuss issues in measurement related to the range of skills assessed, their capacity to measure change over time, and the inclusion of other indicators to measure home, classroom and school factors related to student achievement. They then discuss how ILSAs have built assessment capacity in developing countries and what further changes would be needed to improve their utility for policymakers.

Competitive comparison and PISA bragging rights: sub-national uses of the OECD's PISA in Canada and the USA

“…In particular, international large-scale assessments have played an important role in the formation (or reformation) of education and education policy (Kamens & Benavot, 2011;Kamens & McNeely, 2010;Lockheed & Wagemaker, 2013). Indeed there is much to be gained from exploring how PISA is used within education policy spaces.…”

Steering the National: Exploring the Education Policy Uses of PISA in Spain

“…It is within this terrain that ILSAs provide policy-makers and the public with evidence about the overall status of the education system within an international comparative framework, which in turn raises anxieties and sparks debates about the need for reform. In exploring the political and technical dimensions of policy uses of PISA, the findings suggest ways that PISA acts both as a "whip" and a "thermometer," using a metaphor developed by Lockheed and Wagemaker (2013). In the 2012À2013 reform of Spanish education, led by the conservative government, PISA is utilized as a "whip," pressing forward a reform agenda on the premise that Spain is underperforming in key subjects of mathematics, science, and reading.…”

“…. Despite the rather minimal statistical significance between Spain's PISA results and the OECD average, and the well-noted skepticism about the reporting of average scores and league tables (Lockheed & Wagemaker, 2013; Torney-Purta & Amadeo, 2013), Spain's "below average" scores on PISA have frequently been used both in popular media (Aunión, 2007;Aunión & Barca, 2010;Luzón & Torres, 2012) and as justification for a system overhaul, as evidenced in education policy documents (MECD, 2012(MECD, , 2013a(MECD, , 2013b.…”

Study Design and Evolution, and the Imperatives of Reliability and Validity

“…The monograph featured the proceedings and conclusions of workshops sponsored by the National Academy of Education and shed light on the often misleading interpretation and use of ILSA data. In similar vein, Lockheed and Wagemaker (2013), in addressing the impact that participation in ILSAs has on developing countries highlighted some of the potential perverse effects on policy when data are misinterpreted.…”