Nonequivalence of measurement in latent variable modeling of multigroup data: A sensitivity analysis.

Abstract

MEASUREMENT NON-EQUIVALENCE: A SENSITIVITY ANALYSIS 2In studies of multiple groups of respondents, such as cross-national surveys and cross-cultural assessments in psychological or educational testing, an important methodological consideration is the comparability or "equivalence" of measurement across the groups. Ideally full equivalence would hold, but very often it does not. If non-equivalence of measurement is ignored when it is present, substantively interesting comparisons between the groups may become distorted. We consider this question in multigroup latent variable modeling of multiple-item scales, specifically latent trait models for categorical items. We use numerical sensitivity analyses to examine the nature and magnitude of the distortions in different circumstances, and the factors which affect them. The results suggest that estimates of multigroup latent variable models can be sensitive to assumptions about measurement, in that non-equivalence of measurement does not need to be extreme before ignoring it may substantially affect cross-group comparisons. We also discuss the implications of such findings on the analysis of large comparative studies.

Explaining Measurement Nonequivalence Using Multilevel Structural Equation Modeling

“…However, it is often the case that statistical analyses fail to establish equivalence, particularly scalar equivalence, across the groups studied (i.e., countries, cultures, or time points). Failing to establish equivalence is a serious problem, because comparative analysis that ignores the absence of measurement invariance runs the risk of drawing wrong conclusions (Davidov et al 2014;Kuha and Moustaki 2015;Steenkamp and Baumgartner 1998). Computation of incorrect group means may result in severely biased group rankings (Little 2013;Little, Lindenberger, and Nesselroade 1999;Steinmetz 2011Steinmetz , 2013, and computing erroneous associations in different groups may result in wrong conclusions about relations between variables of interest.…”

“…However, they also involve the risk that such comparisons may be problematic when equivalence is not given. On the one hand, if researchers ignore the finding that concepts are not comparable and continue with substantive comparative work, it could well be the case that findings are biased and conclusions may not be meaningful, as several previous studies have demonstrated (see, e.g., Kuha and Moustaki 2015). On the other hand, if researchers decide in such a case to refrain from any comparisons, it may have the consequence that data are not exhausted and their potential is not realized.…”

The Importance of Isomorphism for Conclusions about Homology: A Bayesian Multilevel Structural Equation Modeling Approach with Ordinal Indicators

“…More recently, general methods have been proposed that examine the consequences of model constraints for particular parameters in models (Kuha and Moustaki, 2015 ; Oberski, 2014 ; Oberski et al, 2015 ) although these approaches are so far untested in the context of isomorphism in multilevel modeling. In this article, we show that absent strong evidence of metric isomorphism, evidence about structural relations across levels is rendered difficult to interpret at best and at worst uninterpretable due to bias in the estimation of structural relations.…”

Adjusting group intercept and slope bias in predictive equations

“…Several options exist when analyzing data that lack FI. Researchers can ignore the problem or estimate non-equivalence models (Kuha & Moustaki, 2015) and argue for valid group comparisons under partial invariance (Steenkamp & Baumgartner, 1998). Ignoring a lack of FI can produce predictive models where one group is unfairly favored and group comparisons are problematic from a theoretical and conceptual perspective.…”

“…Neither ignoring a lack of FI, nor estimating non-equivalent models, can be justified when making decisions about people. Kuha and Moustaki (2015) concluded that the sensitivity of group comparisons under non-equivalent measurement can be severe, leading to biased conclusions. Kim et al (2017) examined five methods for testing groups for measurement invariance.…”