Investigating the Relative Impact of Different Sources of Measurement Non-Equivalence in Comparative Surveys

Roberts, Caroline; Sarrasin, Oriane; Stähli, Michèle Ernst

doi:10.18148/srm/2020.v14i4.7416

Cited by 3 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…. associated with the rejection of statistical measurement invariance (Leitgöb, 2022;Roberts et al, 2020).…”

Section: Measurement Invariancementioning

confidence: 99%

“…Dong and Dumas (2020) report in a meta-analysis that scalar invariance between ethnic groups was not supported for any of the personality inventories considered. One line of research tried to develop less restrictive data analysis methods (Muthén & Asparouhov, 2013;Asparouchov & Muthén, 2014), while the other line of research has been targeting the question as to which circumstances of data collection situation or cognitive respondents' problems are associated with the rejection of statistical measurement invariance (Leitgöb, 2022;Roberts et al, 2020).…”

Section: Measurement Invariancementioning

confidence: 99%

“…Whereas numerous studies report the lack of support for measurement invariance in cross-cultural and other group comparisons, little is understood about problems in questionnaires or data collection situations associated with the comparability bias. It is assumed that the results of measurement invariance analysis can be positively affected if comparability bias is prevented by means of appropriately designed data collection situations or instrumentation (e.g., Leitgöb, 2022;Roberts et al, 2020). Previous research has reported on the lack of support for measurement invariance between different data collection modes, i.e.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Reducing Cross-Cultural Comparability Bias Obtained with Measurement Invariance Analysis by Means of Anchoring Vignettes in Heterogeneous Refugee Samples

Menold

Biddle

Hermanni

et al. 2023

Preprint

View full text Add to dashboard Cite

Background: Configural, metric, and scalar measurement invariance have been indicators of bias-free statistical cross-group comparisons, although they are difficult to verify in the data. Low comparability of translated questionnaires or the different use of response formats by respondents might lead to rejection of measurement invariance and point to comparability bias in studies that use different languages. Anchoring vignettes have been proposed as a method to control for the different use of response formats by respondents (RC-DIF) as implemented by means of rating scales. We evaluate the question whether the comparability bias obtained by means of measurement invariance analysis can be reduced by means of anchoring vignettes or by considering socio-demographic heterogeneity as an alternative approach. Methods: We use the Health System Responsiveness (HSR) questionnaire in English and Arabic in a refugee population. We collected survey data in English (n = 183) and Arabic (n=121) in a random sample of refugees in the third largest German federal state. We conducted multiple-sample Confirmatory Factor Analyses (MGCFA) to analyse measurement invariance and compared the results when 1) using rescaled data on the basis of anchoring vignettes (non-parametric approach), 2) including information on DIF from the analyses with anchoring vignettes as covariates (parametric approach) and 3) including socio-demographic covariates. Results: For the HSR, every level of measurement invariance between Arabic and English questionnaires was rejected. Implementing rescaling on the basis of anchoring vignettes provided superior results over the initial MGCFA analysis, since configural, metric and scalar invariance could not be rejected. When using solely socio-demographic covariates, scalar measurement invariance could not be rejected, but configural and metric invariance had to be rejected. Conclusions: Surveys may consider anchoring vignettes as a method to obtain more satisfactory results of measurement invariance analyses; however, socio-demographic information cannot be included in the models as a standalone method. More research on the efficient implementation of anchoring vignettes and further development of methods to incorporate them when modelling measurement invariance is needed.

show abstract

“…. associated with the rejection of statistical measurement invariance (Leitgöb, 2022;Roberts et al, 2020).…”

Section: Measurement Invariancementioning

confidence: 99%

Section: Measurement Invariancementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Reducing Cross-Cultural Comparability Bias Obtained with Measurement Invariance Analysis by Means of Anchoring Vignettes in Heterogeneous Refugee Samples

Menold

Biddle

Hermanni

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The potential robustness of an instrument regarding structure and item content was put forward as a potential reason, as was the cancelling out of imperfections in both translations (Perneger et al 1999). Roberts et al (2020) assessed a subjective well-being measure across scale formats, modes, as well as linguistic and cultural contexts. They found that translation versions and cultural differences contributed more to non-equivalence than scale format or mode.…”

Section: Introductionmentioning

confidence: 99%

Showcasing the usefulness of web probing: Do subtle variations in questionnaire translation lead to different survey responding?

Behr,

Braun,

Aiglstorfer

2024

Qual Quant

View full text Add to dashboard Cite

It is generally taken for granted that comparability in comparative research hinges, among others, on the quality of questionnaire translations. However, what do slight differences in translation mean for respondents’ answers? In this article, we look at a combination of quantitative evidence from split-ballot experiments and qualitative evidence from additional probing questions for three items that were translated according to different translation methods, resulting in different translations, e.g., for “our national way of life.” Two of the three items do not show any quantitative differences between translation versions when implemented in split-ballot experiments. However, using open-ended probing questions we delved deeper into the effects of different translation versions. This allowed us to show that different translations do indeed change respondent understanding. We suggest mechanisms that may lead to different translations (not) having an impact on the data, and we also try to align the results to the notion of equivalence/comparability in translation. Ultimately, we showcase the usefulness of web probing for exploring different translation understandings.

show abstract

Ensuring cross-cultural data comparability by means of anchoring vignettes in heterogeneous refugee samples

Menold,

Biddle,

von Hermanni

et al. 2023

BMC Med Res Methodol

View full text Add to dashboard Cite

Background Configural, metric, and scalar measurement invariance have been indicators of bias-free statistical cross-group comparisons, although they are difficult to verify in the data. Low comparability of translated questionnaires or the different understanding of response formats by respondents might lead to rejection of measurement invariance and point to comparability bias in multi-language surveys. Anchoring vignettes have been proposed as a method to control for the different understanding of response categories by respondents (the latter is referred to as differential item functioning related to response categories or rating scales: RC-DIF). We evaluate the question whether the cross-cultural comparability of data can be assured by means of anchoring vignettes or by considering socio-demographic heterogeneity as an alternative approach. Methods We used the Health System Responsiveness (HSR) questionnaire and collected survey data in English (n = 183) and Arabic (n = 121) in a random sample of refugees in the third largest German federal state. We conducted multiple-group Confirmatory Factor Analyses (MGCFA) to analyse measurement invariance and compared the results when 1) using rescaled data on the basis of anchoring vignettes (non-parametric approach), 2) including information on RC-DIF from the analyses with anchoring vignettes as covariates (parametric approach) and 3) including socio-demographic covariates. Results For the HSR, every level of measurement invariance between the Arabic and English languages was rejected. Implementing rescaling or modelling on the basis of anchoring vignettes provided superior results over the initial MGCFA analysis, since configural, metric and – for ordered categorical analyses—scalar invariance could not be rejected. A consideration of socio-demographic variables did not show such an improvement. Conclusions Surveys may consider anchoring vignettes as a method to assess cross-cultural comparability of data, whereas socio-demographic variables cannot be used to improve data comparability as a standalone method. More research on the efficient implementation of anchoring vignettes and further development of methods to incorporate them when modelling measurement invariance is needed.

show abstract

Investigating the Relative Impact of Different Sources of Measurement Non-Equivalence in Comparative Surveys

Cited by 3 publications

References 0 publications

Reducing Cross-Cultural Comparability Bias Obtained with Measurement Invariance Analysis by Means of Anchoring Vignettes in Heterogeneous Refugee Samples

Reducing Cross-Cultural Comparability Bias Obtained with Measurement Invariance Analysis by Means of Anchoring Vignettes in Heterogeneous Refugee Samples

Showcasing the usefulness of web probing: Do subtle variations in questionnaire translation lead to different survey responding?

Ensuring cross-cultural data comparability by means of anchoring vignettes in heterogeneous refugee samples

Contact Info

Product

Resources

About