The potential of anchoring vignettes to increase intercultural comparability of non-cognitive factors

An Examination of Different Scale Usage Correction Procedures to Enhance Cross-Cultural Data Comparability

“…Similar conclusions were obtained from studies targeting design-based correction procedures applied in the 2012 Programme for International Student Assessment (PISA) data. In these studies, design-based corrections did not always result in scalar invariance (e.g., Marksteiner et al, 2019), and the relations with the criterion (i.e., achievement) differed between uncorrected and corrected scores (e.g., Vonkova, Papajoanu, et al, 2018). One limitation of this work is that each correction procedure was independently assessed, with the assumption that such corrections operate on data from all respondents to the same extent.…”

“…We assume that if any of the abovementioned procedures enhances the comparability of cross-cultural Likert-scale data, corrected scores should show a higher level of measurement invariance than raw scores. Two applications of anchoring vignettes in the PISA data from 2012 showed enhanced measurement invariance of scales across dozens of countries (He, Buchholz, et al, 2017; Marksteiner et al, 2019). However, anchoring vignettes did not result in scalar invariance (e.g., the mean scores of rescaled data cannot be compared validly across cultures) in all of the target scales in these studies.…”

Assessment design and quality of inferences in PISA: limitations and recommendations for improvement

“…The articles by Marksteiner, Kuger, and Klieme (2018) and He, Barrera-Pedemonte, and Buchholz (2018) deal with cross-cultural comparability issues of non-cognitive PISA data. Extensive research has examined measurement invariance across cultures of student achievement but less evidence is available for other constructs measured in international assessments.…”

“…positive association between student motivation and student achievement at student level and negative at country level). Marksteiner et al (2018) analyse the potential of anchoring vignettes for improving cross-cultural comparability of non-cognitive student data. PISA has used anchoring vignettes in student questionnaires.…”

“…Ratings of students on these descriptions can be used as standards to adjust self-reported responses for different interpretations of identical questions across cultures ex-post. Marksteiner et al (2018) use student-organisation-related data from the PISA 2015 field trial for 51 countries to evaluate and compare the reliability of anchored and non-anchored data. Reliability analysis included measurement invariance, internal consistency, and exploratory factor analysis for scale dimensionality whenever internal consistency was low.…”