Evidence of Validity Does not Rule out Systematic Bias: A Commentary on Nomological Noise and Cross-Cultural Invariance

Abstract

We comment on the argument by Welzel, Brunkert, Kruse and Inglehart (2021) that theoretically expected associations in nomological networks should take priority over invariance tests in cross-national research. We agree that narrow application of individual tools, such as multi-group confirmatory factor analysis with data that violates the assumptions of these techniques, can be misleading. However, findings that fit expectations of nomological networks may not be free of bias. We present supporting evidence of systematic bias affecting nomological network relationships from a) previous research on intelligence and response styles, b) two simulation studies, and c) data on the choice index from the World Value Survey (WVS). Our main point is that nomological network analysis by itself is insufficient to rule out systematic bias in data. We point out how a thoughtful exploration of sources of biases in cross-national data can contribute to stronger theory development.

Individualism-Collectivism: A Brazilian-Portuguese Version of Triandis and Gelfand’s Scale

“…To achieve this, we conducted CFA and invariance analysis in the second half, also composed of the USA and Brazilian sample and compared their structure, estimated by maximum likelihood. To do so, we used R's statistical packages such as lavaan, semTool, and semPlot, tested the original structure and the structure suggested by the EFA, and then, we tested modification indices and invariance measures (Fischer et al, 2022;Hirschfeld & Von Brachel, 2014;Jorgensen et al, 2018;Lugtig & Hox, 2012;Milfont & Fischer, 2010;Rosseel, 2014;Smith et al, 2013;Tomás et al, 2014) and considered one way to conduct measurement invariance (Van de Schoot et al, 2015).…”

“…The study 1 method applied decentering translation, exploratory factor analysis (EFA), confirmatory factor analysis (CFA) with modification indices and equivalence tests (Davidson & Thomson, 1980;Fischer et al, 2022;Hirschfeld & Von Brachel, 2014;Milfont & Fischer, 2010;Neilands et al, 2018;Smith et al, 2013).…”

“…Thus, this work aims to empirically test the stability of the HVIC structure in Brazil, using the short 16-item version of Triandis and Gelfand (1998) scale. Firstly, we sought to replicate the Triandis and Gelfand original scale in both a USA and a Brazilian sample, analyzing invariance across nations (Fischer et al, 2022). In addition, we compared two student samples-one from Brazil and the other from the USA (study 1).…”

Niche diversity effects on personality measurement – evidence from large national samples during the COVID-19 pandemic

“…Arguably, using the factor replicability is a more appropriate test of the niche construction hypothesis, because it provides an overall estimate on the organization of the personality space as inferred from the survey responses instead of assuming that the factors are present and then only examining the relative relation of factors to each other. We strongly encourage future studies to consider the cognitive implications at the psychometric level instead of just analyzing the factor interrelations because in the face of substantiative factor incomparability these intercorrelations might not be meaningful ( Fischer et al, 2022 ; Lasker et al, 2022 ).…”

Distinct Conceptions of Freedom in East Asia and the Protestant West Underpin Unique Pathways of Societal Development

“…Therefore, this is an ecological level analysis that follows an "ecologic," which may differ from that of individuals within societies (Leung & Bond, 2015). It has been disputed whether measurement invariance tests are a necessary and adequate procedures for confirming the cross-cultural comparability of such constructs (see, e.g., Fischer et al, 2022;Meuleman et al, 2022;Welzel et al, 2021). Supplemental Appendix, Part D, discusses the issue of measurement invariance.…”