What PISA Intends to and Can Possibly Achieve: A Critical Programme Theory Analysis

Abstract

This paper advances the enlightened discussion of the nature, logic, and possible effects of the Programme for International Student Assessment (PISA). The purpose is to analyze the assumptions regarding how PISA is to achieve its intended effects, that is, to reconstruct PISA's programme theory (PT) and to probe the validity of its underlying assumptions. The paper demonstrates that PISA's PT has low internal validity. However, some PISA assumptions are consistent, for example, the assumption that legitimization activities justify PISA as a transnational benchmarking system measuring education system performance.PISA exemplifies systemic evaluation governance: all actors in the field are expected to use PISA results to react to and reflect on their own practice, compare themselves with others, and then act accordingly to improve education systems and school practice, though no activities or resources are allocated to change school practice. There is no empirical research into how systemic evaluation governance works in practice that can be used to probe PISA's external validity. PISA's PT is in line with discourse policy, governance theory, and school effectiveness research, but whether and under what conditions and how PISA helps change education systems and school practice are empirical questions waiting to be answered.

Evaluation systems in a crowded policy space: Implications for local school governance

“…The Programme for International Student Assessment (PISA) is a major international student test developed by the OECD to provide policy actors at various levels with trustworthy performance data that can be used for evaluating and benchmarking education systems (Grek 2012;Hanberger 2014;Mangez and Hilgers 2012). PISA is regarded as the key pillar in producing knowledge used to shape educational policy in many countries (Carvalho 2012).…”

“…all school actors should consider their responsibility for their country's performance on PISA. The OECD provides knowledge and recommendations for improving education systems in line with the OECD's mission of striving to minimise social disparities in education and economic growth (Hanberger 2014).…”

“…Although PISA does not provide results on how schools perform at the local or individual school levels, municipal education committees, principals and teachers are supposed to take the national PISA scores and the OECD's policy recommendations into account when developing their schools (Hanberger 2014).…”

“…That several evaluation systems point in the same direction adds to the scaling down of the school law and curriculum and, in doing so, leaves little or no discretion to local school governance. With its emphasis on an even narrower range of aspects of education that to a very limited extent capture the goals of the Swedish education system, and by putting pressure on national and local actors to promote reforms and programmes in line with an OECD-oriented education discourse (Hanberger 2014), PISA adds to the described 'policy by the way'.…”

“…In our view, it is questionable whether grades and tests are valid measures of performance at the school, municipal and national levels. The assumption that OECD/PISA is a valid test for measuring the quality of education systems can also be questioned (Hanberger 2014). Well-founded evidential criticism of the relevance of using student grades and national tests for various purposes can, furthermore, be found in other studies.…”