A Mixture Model for Random Responding Behavior in Forced-Choice Noncognitive Assessment: Implication and Application in Organizational Research

Abstract

For various reasons, respondents to forced-choice assessments (typically used for noncognitive psychological constructs) may respond randomly to individual items due to indecision or globally due to disengagement. Thus, random responding is a complex source of measurement bias and threatens the reliability of forced-choice assessments, which are essential in high-stakes organizational testing scenarios, such as hiring decisions. The traditional measurement models rely heavily on nonrandom, construct-relevant responses to yield accurate parameter estimates. When survey data contain many random responses, fitting traditional models may deliver biased results, which could attenuate measurement reliability. This study presents a new forced-choice measure-based mixture item response theory model (called M-TCIR) for simultaneously modeling normal and random responses (distinguishing completely and incompletely random). The feasibility of the M-TCIR was investigated via two Monte Carlo simulation studies. In addition, one empirical dataset was analyzed to illustrate the applicability of the M-TCIR in practice. The results revealed that most model parameters were adequately recovered, and the M-TCIR was a viable alternative to model both aberrant and normal responses with high efficiency.

Detecting Careless Responding in Multidimensional Forced-Choice Questionnaires

“…To 1 Currently, model-based approaches to detect careless responding in rating scale data at the item level are being developed (e.g., Ulitzsch et al, 2021). Even though there are already first implementations for the forced-choice format (e.g., Peng et al, 2023), we focus on the development of indices that come with the major advantage that they are easy to use for practitioners. combine these two approaches, we extended the response time effort index developed by Wise and Kong (2005) to include an upper threshold for delayed responses (𝑆 𝑢𝑝𝑝𝑒𝑟 ), yielding the RTI for the MFC format.…”