On Detecting Systematic Measurement Error in Cross-Cultural Research: A Review and Critical Reflection on Equivalence and Invariance Tests

Abstract

One major threat to revealing cultural influences on psychological states or processes is the presence of bias (i.e., systematic measurement error). When quantitative measures are not targeting the same construct or they differ in metric across cultures, the validity of inferences about cultural variability (and universality) is in doubt. The objectives of this article are to review what can be done about it and what is being done about it. To date, a multitude of useful techniques and methods to reduce or assess bias in cross-cultural research have been developed. We explore the limits of invariance/equivalence testing and suggest more flexible means of dealing with bias. First, we review currently available established and novel methods that reveal bias in cross-cultural research. Second, we analyze current practices in a systematic content analysis. The content analysis of more than 500 culture-comparative quantitative studies (published from 2008 to 2015 in three outlets in cross-cultural, social, and developmental psychology) aims to gauge current practices and approaches in the assessment of measurement equivalence/invariance. Surprisingly, the analysis revealed a rather low penetration of invariance testing in cross-cultural research. Although a multitude of classical and novel approaches for invariance testing is available, these are employed infrequent rather than habitual. We discuss reasons for this hesitation, and we derive suggestions for creatively assessing and handling biases across different research paradigms and designs.

Fatalism as a traditional cultural belief potentially relevant to trauma sequelae: Measurement equivalence, extent and associations in six countries

“…Testing of invariance followed standardized procedures and conventions for cross-cultural comparisons (see Boer, Hanke, & He, 2018). In the first step that tested configural invariance, confirmatory factor analyses (CFAs) were conducted separately for each country in order to confirm the latent model structure.…”

“…Regarding measurement invariance testing, differences in CFI and RMSEA were compared between the more restricted and its previous model (ΔCFI and ΔRMSEA). Conventions suggest ΔCFI ≤ .02 and ΔRMSEA ≤ .03 from configural to weak invariance and ≤ .01 for both indices from weak to strong invariance (Boer et al, 2018). The effect size Cohen's d was calculated by applying the formula described in Choi, Fan, and Hancock (2009).…”

Endogenous Institutions and Economic Outcomes

“…Even if focusing on the items most consistent with the underlying concept of a ‘culture of cooperation’ is key to improving the precision of the estimation, OLS estimates are still bound to be greatly attenuated since self‐reported norms of conduct ‘are not targeting the same construct or they differ in metric’ (Boer et al . , p. 713) across cultures. For instance, measuring culture in increasingly market‐oriented societies might induce cognitive dissonance or self‐serving biases because of the liberal and open‐minded ideals of the respondents.…”

A meeting report: cross-cultural comparability of questionnaire measures in large-scale international surveys

“…Perhaps more concerning is the fact that the most frequent practice is, in fact, to simply ignore the possible non-equivalence of measurement in cross-cultural research: many secondary users of the data compare respondents' answers and scale values derived from statistical models without acknowledging, and discussing, the potential threats to comparability (Boer, Hanke, & He, 2018). Other scholars resort to generalisations based on the analysis of single items-a situation in which comparability of measurements cannot be formally assessed based on the properties of a measurement model.…”