Analysis of Cross-Cultural Comparability of PISA 2009 Scores

Abstract

General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.-Users may download and print one copy of any publication from the public portal for the purpose of private study or research -You may not further distribute the material or use it for any profit-making activity or commercial gain -You may freely distribute the URL identifying the publication in the public portal Take down policyIf you believe that this document breaches copyright, please contact us providing details, and we will remove access to the work immediately and investigate your claim. AbstractThe Program for International Student Assessment (PISA) is a large-scale cross-national study that measures academic competencies of 15-year-old students in mathematics, reading, and science from more than 50 countries/economies around the world. PISA results are usually aggregated and presented in so-called "league tables," in which countries are compared and ranked in each of the three scales. However, to compare results obtained from different groups/countries, one must first be sure that the tests measure the same competencies in all cultures. In this paper, this is tested by examining the level of measurement equivalence in the 2009 PISA data set using an item response theory approach (IRT) and analyzing differential item functioning (DIF). Measurement in-equivalence was found in the form of uniform DIF. In-equivalence occurred in a majority of test questions in all three scales researched and is, on average, of moderate size. It varies considerably both across items and across countries. When this uniform DIF is accounted for in the in-equivalent model, the resulting country scores change considerably in the cases of the "Mathematics," "Science," and especially, "Reading" scale. These changes tend to occur simultaneously and in the same direction in groups of regional countries. The most affected seems to be Southeast Asian countries/territories whose scores, although among the highest in the initial, homogeneous model, additionally increase when accounting for in-equivalence in the scales.

Effects of Anchoring Vignettes on Comparability and Predictive Validity of Student Self-Reports in 64 Cultures

“…comparability and predictive validity in the cognitive assessment (e.g., Kankaraš & Moors, 2014) and self-report data (e.g., van de Gaer, Grisay, Schulz, & Gebhardt, 2012) may have prevented full-fledged explorations of these data for basic research and evidence-based policy making (Goldstein, 2004;Gorur, 2014). Innovative designs of item formats and sophisticated psychometric methods promise to ameliorate such concerns in self-reports (Kyllonen & Bertling, 2014).…”

Uncertainty and Sensitivity Analyses of PISA Efficiency: Distance Based Analysis Approach

“…PISA focuses on young people's ability to use their knowledge and skills to meet real-life challenges [23,24,26]. This orientation reflects the change in the goals and objectives of curricula themselves.…”

“…PISA includes comparative international tests of students' achievements. It has become increasingly popular over the last ten years [22], with a major impact on the educational community, public, and national and international policy makers [23]. The relevance of knowledge and skills measured by PISA is confirmed by recent studies [23] that have been keeping track of the young people in years after they were assessed.…”

“…It has become increasingly popular over the last ten years [22], with a major impact on the educational community, public, and national and international policy makers [23]. The relevance of knowledge and skills measured by PISA is confirmed by recent studies [23] that have been keeping track of the young people in years after they were assessed. Studies in Australia, Canada, and Denmark show a strong interconnection between a student's performance in reading and a chance to complete secondary school and carry on with post-secondary studies at the age 19 [3].…”

“…There are countries that share similarities in education, such as universal public systems and diversity in student population. Cross-cultural studies on PISA results [23,24,25,26] find that some are performing at the top level while some are average performing countries [24]. According to Lounkaew [27], the differences in achievements between urban and rural students can be explained by intangible and immeasurable school characteristics.…”