Mixed-Keying or Desirability-Matching in the Construction of Forced-Choice Measures? An Empirical Investigation and Practical Recommendations

Abstract

Forced-choice (FC) measures are becoming increasingly popular as an alternative to single-statement (SS) measures. However, to ensure the practical usefulness of an FC measure, it is crucial to address the tension between psychometric properties and faking resistance by balancing mixed keying and social desirability matching. It is currently unknown from an empirical perspective whether the two design criteria can be reconciled, and how they impact respondent reactions. By conducting a two-wave experimental design, we constructed four FC measures with varying degrees of mixed-keying and social desirability matching from the same statement pool and investigated their differences in terms of psychometric properties, faking resistance, and respondent reactions. Results showed that all FC measures demonstrated comparable reliability and induced similar respondent reactions. Forced-choice measures with stricter social desirability matching were more faking resistant, while FC measures with more mixed keyed blocks had higher convergent validity with the SS measure and displayed similar discriminant and criterion-related validity profiles as the SS benchmark. More importantly, we found that it is possible to strike a balance between social desirability matching and mixed keying, such that FC measures can have adequate psychometric properties and faking resistance. A seven-step recommendation and a tutorial based on the autoFC R package were provided to help readers construct their own FC measures.

Exploring the Role of Social Desirability in Situational Judgment Tests

“…This may help maximize internal consistency, but efforts are not usually made to balance the response options on social desirability or other factors besides trait level. This is important to note given that desirability matching is key to minimizing faking in forced-choice personality assessment (Li et al, 2024) and desirability appears to play a role in how individuals respond to SJT items (Reznik et al, 2023). Therefore, we include both skill and trait-focused SJTs in our study given that there is little prior research comparing these tests.…”

“…Although prior SJT research has focused on the situational context by quantifying the strength of the situation (Freudenstein et al, 2024), our results suggest that it may be more important to focus on the strength and variability among response options. We argue that future SJT research should adopt practices from forced-choice personality measurement where attempts are taken to maximize trait measurement by matching response alternatives with similar levels of social desirability (Edwards, 1953;Li et al, 2024). To our knowledge, researchers or practitioners are not likely to measure these other factors when creating response option sets for SJT items.…”