The Impact of Different Methods to Correct for Response Styles on the External Validity of Self-Reports

Abstract

Abstract. Response styles (RSs) such as acquiescence represent systematic respondent behaviors in self-report questionnaires beyond the actual item content. They distort trait estimates and contribute to measurement bias in questionnaire-based research. Although various approaches were proposed to correct the influence of RSs, little is known about their relative performance. Because different correction methods formalize the latent traits differently, it is unclear how model choice affects the external validity of the corrected measures. Therefore, the present study on N = 1,000 Dutch respondents investigated the impact of correcting responses to measures of self-esteem and the need for cognition using structural equation models with structured residuals, multidimensional generalized partial credit models, and multinomial processing trees. The study considered three RSs: extreme, midpoint, and acquiescence RS. The results showed homogeneous correlation patterns among the modeled latent and external variables, especially if they were not themselves subject to RSs. In that case, the IRT-based models, including an uncorrected model, still yielded consistent results. Nevertheless, the strength of the effect sizes showed variation.

Reliability and Factorial Validity of the Core Self-Evaluations Scale

“…For example, in online research C/IER responses can bias assessments (Woods, 2006). Thus, empirical research is well-advised to adopt appropriate countermeasures, for example, by excluding conspicuous respondents (e.g., Arias et al, 2022; Schroeders et al, 2022) or explicitly modeling response styles (e.g., Aichholzer, 2014; Scharl & Gnambs, 2022). Finally, although the reported results do not exempt researchers from conducting systematic analyses of measurement invariance before addressing substantive research questions, our findings provide preliminary evidence of comparable measurement structures in English, German, and Spanish versions of the CSES.…”

“…Finally, an important extension of the present findings would be meta-analytic research on the criterion validity of the CSES. In this context, it might also be worthwhile to evaluate whether these validity correlations are susceptible to the choice of a specific psychometric model or how response styles are taken into account (see Scharl & Gnambs, 2022, for related findings).…”

Reliability and factorial validity of the Core Self-Evaluations Scale: A meta-analytic investigation of wording effects

“…Finally, an important extension of the present findings would be metaanalytic research on the criterion validity of the CSES. In this context, it might also be worthwhile to evaluate whether these validity correlations are susceptible to the choice of a specific artifact model and how response styles are modeled (see Scharl & Gnambs, 2022, for related findings).…”