The comparability of the universalism value over time and across countries in the European Social Survey: exact vs. approximate measurement invariance

Abstract

Over the last decades, large international datasets such as the European Social Survey (ESS), the European Value Study (EVS) and the World Value Survey (WVS) have been collected to compare value means over multiple time points and across many countries. Yet analyzing comparative survey data requires the fulfillment of specific assumptions, i.e., that these values are comparable over time and across countries. Given the large number of groups that can be compared in repeated cross-national datasets, establishing measurement invariance has been, however, considered unrealistic. Indeed, studies which did assess it often failed to establish higher levels of invariance such as scalar invariance. In this paper we first introduce the newly developed approximate approach based on Bayesian structural equation modeling (BSEM) to assess cross-group invariance over countries and time points and contrast the findings with the results from the traditional exact measurement invariance test. BSEM examines whether measurement parameters are approximately (rather than exactly) invariant. We apply BSEM to a subset of items measuring the universalism value from the Portrait Values Questionnaire (PVQ) in the ESS. The invariance of this value is tested simultaneously across 15 ESS countries over six ESS rounds with 173,071 respondents and 90 groups in total. Whereas, the use of the traditional approach only legitimates the comparison of latent means of 37 groups, the Bayesian procedure allows the latent mean comparison of 73 groups. Thus, our empirical application demonstrates for the first time the BSEM test procedure on a particularly large set of groups.

The Index of Emancipative Values: Measurement Model Misspecifications

“…Second, there is still a lack of evidence from simulation studies about how small should be the prior variance on differences in factor loadings/intercepts across groups to conclude with certainty that the approximate invariance does hold . The existing results suggest that the prior variance of 0.05 is sufficient to avoid substantial bias in latent mean comparisons (Van de Schoot et al, 2013), and few practical applications of the method, including one in the presented paper, rely on that recommended cut-off value (e.g., Zercher et al, 2015. I tried to impose stricter levels of invariance for each of the six wave-specific models.…”

“…In addition, one can simply remove the most outlying unit(-s) from the sample or try to establish partial (metric or scalar) approximate invariance by releasing approximate equality constraints for problematic items (that is, loadings and/or intercepts for which the highest countryspecific deviations from the sample average parameter values are observed). 16 I do not apply frequentist MGCFA to the same samples, as it was done [for the purpose of comparison between the two approaches] in similar applications of the Bayesian approximate invariance testing by Davidov et al (2015), Cieciuch et al (2014), and Zercher et al (2015). I have already shown that the classical approach failed to establish scalar invariance of -choice‖ even across ten cultural zones, so it is quite reasonable to anticipate that it will fail at the country level (much more heterogeneous) as well.…”

“…B3 about hereIf a model with some acceptable level of invariance (defined by prior variance on parameter differences) fits sufficiently well, one can conclude that approximate invariance holds in the data. Both simulation studies and empirical research suggest that the prior variance on between-group parameter differences as high as 0.05 does not introduce substantial bias in the estimates of latent means and therefore allows for meaningful comparisons across groups(Muthen and Asparouhov 2013; van de Shoot et al 2013;Davidov et al 2015;Cieciuch et al 2014;Zercher et al 2015). The approximate Bayesian approach to invariance testing has been shown in a few recent papers to be quite flexible in handling relatively small (but numerous) between-group differences in model parameters.…”

A Monte Carlo Simulation Study to Assess The Appropriateness of Traditional and Newer Approaches to Test for Measurement Invariance

“…3 For partial exact invariance conditions, we considered situations in which 25%, 50%, 75%, and 100% of the compared groups were countries. However, one should keep in mind that if the focus of research is on comparisons both across countries and time points simultaneously, the number of groups may quickly increase to 90 and more (Marsh et al, 2017;Zercher et al, 2015).…”

Explaining Measurement Nonequivalence Using Multilevel Structural Equation Modeling

“…Approximate measurement invariance allows small differences in loadings or intercepts across groups. Moreover, testing for approximate measurement invariance can be performed within the Bayesian SEM framework Muthén and Asparouhov 2013;van de Schoot et al 2013;Zercher et al 2015).…”