On Bank Assembly and Block Selection in Multidimensional Forced-Choice Adaptive Assessments

Abstract

Multidimensional forced-choice (FC) questionnaires have been consistently found to reduce the effects of socially desirable responding and faking in noncognitive assessments. Although FC has been considered problematic for providing ipsative scores under the classical test theory, item response theory (IRT) models enable the estimation of nonipsative scores from FC responses. However, while some authors indicate that blocks composed of opposite-keyed items are necessary to retrieve normative scores, others suggest that these blocks may be less robust to faking, thus impairing the assessment validity. Accordingly, this article presents a simulation study to investigate whether it is possible to retrieve normative scores using only positively keyed items in pairwise FC computerized adaptive testing (CAT). Specifically, a simulation study addressed the effect of (a) different bank assembly (with a randomly assembled bank, an optimally assembled bank, and blocks assembled on-the-fly considering every possible pair of items), and (b) block selection rules (i.e., T, and Bayesian D and A-rules) over the estimate accuracy and ipsativity and overlap rates. Moreover, different questionnaire lengths (30 and 60) and trait structures (independent or positively correlated) were studied, and a nonadaptive questionnaire was included as baseline in each condition. In general, very good trait estimates were retrieved, despite using only positively keyed items. Although the best trait accuracy and lowest ipsativity were found using the Bayesian A-rule with questionnaires assembled on-the-fly, the T-rule under this method led to the worst results. This points out to the importance of considering both aspects when designing FC CAT.

An Item Response Theory Model for Incorporating Response Times in Forced-Choice Measures

“…As stated by Hicks (1970), ipsative scores are expected to provide negatively biased trait intercorrelations, and the average validity of the ipsative trait scores with an external criterion is expected to be zero. Therefore, the bias in the correlations of trait scores (Φ-Bias) and the average correlation between the estimated trait scores and a simulated external criterion ξ (denoted by r ¯ ξ η ^ ) were computed to assess the ipsativity of estimated trait scores (Kreitchmann et al, 2023). For the calculation of Φ-Bias and r ¯ ξ η ^ , Fisher’s z transformation was used to convert the original correlation coefficients to Fisher’s z prior to calculating correlation differences or means across all trait scores, then back-transformed to the correlation metric (e.g., Silver & Dunlap, 1987).…”

“…First, we consider three different test designs: (a) all item pairs using positively keyed items, (b) half of the item pairs using positively keyed items, and the other half of item pairs using a mixture of positively and negatively keyed items, and (c) all item pairs using positively keyed items with optimal design., that is, make the difference between the two scale parameters of an item pair as large as possible. Several studies have suggested that such an optimal design would be helpful to parameter recovery of the Thurstonian model (Bürkner, 2022; Kreitchmann et al, 2023). The number of item pairs in these three designs was 12 or 24.…”