Enhancing the Cross-Cultural Comparability of Self-Reports Using the Overclaiming Technique: An Analysis of Accuracy and Exaggeration in 64 Cultures

Abstract

The overclaiming technique (OCT) is a novel way of measuring how socially desirable responding influences survey responses. It has the potential to enhance the cross-cultural comparability of survey data. It allows the identification of respondents’ knowledge accuracy and exaggeration by comparing their assessments of familiarity with existing and nonexisting concepts in a particular field of knowledge. We aim to compare the response patterns of countries and world regions based on their OCT accuracy and exaggeration-index values and validate these OCT scores using external variables. We also introduce a general model for the categorization of respondents based on their OCT indices values. We use the Programme for International Student Assessment (PISA) 2012 data from 64 countries (N = 275,904). We found considerable differences in response patterns across world regions: high accuracy values in East Asia, low accuracy values in Southern and Central America, high exaggeration values in Southern Europe, and low exaggeration values in Western Europe. Furthermore, we show that familiarity with math concepts changes substantially after adjustment using the OCT. The correlation between unadjusted math familiarity and math test scores is weak and nonsignificant (.13) whereas after adjustment the correlation becomes strong and significant (.66). Concerning other indicators such as gross domestic product (GDP), public expenditure in education, and Corruption Perceptions Index (CPI), before adjustment the correlation is negative and even significant for CPI (–.11, –.22, and –.45) whereas after adjustment the correlation becomes, though nonsignificant, positive (.05, .19, and .07). We also discuss the OCT in the context of other methods indicative of culturally preferred scale usage.

Exploring the effect of lexical inferencing and dictionary consultation on undergraduate EFL students’ vocabulary acquisition

“…However, there are cross-cultural differences in accurately estimating and overestimating one’s knowledge. For example, Vonkova et al [ 99 ] found that students in East Asia estimate their knowledge highly accurately, but students in Southern and Central America do not. Furthermore, students in Southern Europe tend to exaggerate their knowledge more than those in Western Europe.…”

“…However, there seems to be no consistent pattern in terms of accurately estimating and overestimating knowledge across the Middle East. Since Saudi Arabia is not included in Vonkova et al’s study [ 99 ], it is not clear how students with this particular cultural background would fare in terms of accuracy and exaggeration of estimated knowledge.…”

An Examination of Different Scale Usage Correction Procedures to Enhance Cross-Cultural Data Comparability

“…Similar conclusions were obtained from studies targeting design-based correction procedures applied in the 2012 Programme for International Student Assessment (PISA) data. In these studies, design-based corrections did not always result in scalar invariance (e.g., Marksteiner et al, 2019), and the relations with the criterion (i.e., achievement) differed between uncorrected and corrected scores (e.g., Vonkova, Papajoanu, et al, 2018). One limitation of this work is that each correction procedure was independently assessed, with the assumption that such corrections operate on data from all respondents to the same extent.…”

“…Applications of the aforementioned procedures are common, and their effects can result in changes in psychometric properties of target constructs, criterion and predictive validity of scales (i.e., structural relations) and mean comparisons across groups (e.g., Bartram, 1996; Vonkova, Papajoanu, et al, 2018). We argue that correction effects on structural relations and mean comparisons should be built on sound psychometric properties of corrected scores.…”

“…Yet, due to the lack of validity evidence using measures other than self-reports, it is difficult to determine if corrected scores are unequivocally more valid. For instance, when overclaiming in math familiarity was corrected for in the PISA data, the weak and nonsignificant correlation between math familiarity and math achievement became strong and significant in 64 countries (Vonkova, Papajoanu, et al, 2018). Additionally, after performing an anchoring vignette correction on the Classroom Management scale in the PISA data, it was revealed that there may be different implicit standards of self-assessment across countries, and this resulted in substantial change in correlations with students test scores and public expenditure per pupil (Vonkova, Zamarro, et al, 2018).…”