Causal Inference on Education Policies: A Survey of Empirical Studies Using Pisa, Timss and Pirls

Abstract

The identification of the causal effects of educational policies is the top priority in recent education economics literature. As a result, a shift can be observed in the strategies of empirical studies. They have moved from the use of standard multivariate statistical methods, which identify correlations or associations between variables only, to more complex econometric strategies, which can help to identify causal relationships. However, exogenous variations in databases have to be identified in order to apply causal inference techniques. This is a far from straightforward task. For this reason, this paper provides an extensive and comprehensive overview of the literature using quasiexperimental techniques applied to three well-known international large-scale comparative assessments, such as PISA, PIRLS or TIMSS, over the period 2004-2016. In particular, we review empirical studies employing instrumental variables, regression discontinuity designs, difference in differences and propensity score matching to the above databases. Additionally, we provide a detailed summary of estimation strategies, issues treated and profitability in terms of the quality of publications to encourage further potential evaluations. The paper concludes with some operational recommendations for prospective researchers in the field.

Curriculum Reform in the Decentralization of Education in Indonesia: Effect on Students’ Achievements

“…In this study, propensity score analysis is conducted to estimate the effect of curriculum reform as part of education decentralization on students' performances in secondary level. The analysis is one technique of causal effects' estimations that allow valid casual inference based on defining the counterfactual group through a quasi-experiment on observational data (Cordero, Cristóbal, & Santín, 2018). With respect to observed covariates, both propensity score models were administered to reduce the selection bias by balancing the treatment and the control groups to look like the full sample (Adelson, Guo & Fraser, 2014;Rosenbaum & Rubin, 1983;Xie, Brand, & Jann, 2012).…”

Meta-Analyzing Individual Participant Data From Studies With Complex Survey Designs: A Tutorial on Using the Two-Stage Approach for Data From Educational Large-Scale Assessments

“…In these searches, we looked for documents that reported the application of methods to data from four major ELSAs: NAEP, PISA, TIMSS, and PIRLS. We chose these ELSAs because their IPD have been widely used in secondary data analyses (Drent et al, 2013;Hernández-Torrano & Courtney, 2021;Hopfenbeck et al, 2018;Lenkeit et al, 2015;Reilly et al, 2015, there is extensive coverage of the results of these ELSAs in the media (Hopfenbeck & Görgen, 2017;Johansson, 2016;Steiner-Khamsi et al, 2018), and the results of these ELSAs are intended to inform educational policies (Beaton & Zwick, 1992) and are also likely to affect such policies (Cordero et al, 2018;Kirsch & Braun, 2020;Tobin et al, 2016). The relevance of these ELSAs was, for example, also emphasized by Kirsch and Braun (2020), who conclude that "assessments such as TIMSS, PIRLS, and PISA are arguably the largest and most widely discussed comparative international assessments" (p. 5).…”

‘Utility’ of education and the role of transformative agency: Policy challenges and agendas

“…Still, human capital logics are also influential for policy decisions on the school level. For example, it was in schools and not tertiary education where the first attempts at international assessment of practical skills (often interpreted as the quality of human capital) took place: the well-known Programme for International Student Assessment (PISA), Trends in International Mathematics and Science Study (TIMSS) and Programme in International Reading Literacy Study (PIRLS) initiatives (see Cordero et al, 2018). According to a World Bank report, almost two-thirds of all data used for assessing the quality of human capital at the international level concerns secondary school scores (see Angrist et al, 2019: 14).…”