Adaptive testing with the GGUM-RANK multidimensional forced choice model: Comparison of pair, triplet, and tetrad scoring

Abstract

Likert-type measures have been criticized in psychological assessment because they are vulnerable to response biases, including central tendency, acquiescence, leniency, halo, and socially desirable responding. As an alternative, multidimensional forced choice (MFC) testing has been proposed to address these concerns. A number of researchers have developed item response theory (IRT) models for MFC data and have examined latent trait estimation with tests of different dimensionality and length. Research has also explored the advantages of computerized adaptive testing (CAT) with MFC pair tests having as many as 25 dimensions, but there have been no published studies on CAT with MFC triplets or tetrads. Thus, in this research we aimed to address that issue. We used recently developed item information functions for an MFC ranking model to compare the benefits of CAT with MFC pair, triplet, and tetrad tests. A simulation study showed that CAT substantially outperformed nonadaptive testing for latent trait estimation across MFC formats. More importantly, CAT with MFC pairs provided estimation accuracy similar to or better than that from tests of equivalent numbers of nonadaptive MFC triplets. On the basis of these findings, implications and recommendations are further discussed for constructing MFC measures to use in psychological contexts.

Computerized Adaptive Testing for Ipsative Tests with Multidimensional Pairwise-Comparison Items: Algorithm Development and Applications

“…For example, Mulder and van der Linden (2009) successfully developed the Fisher information (FI) based criteria, such as the optimal determinant (D-optimality) and trace of information matrix (A-optimality), to select items in multidimensional CAT. Although certain pioneering studies revisited these algorithms for IT-CAT (Chen et al, 2020; Joo et al, 2020), further investigation is warranted.…”

“…Unlike many CAT studies for normative tests, thus far, few studies have addressed the development of the IT-CAT. To the best of our knowledge, there are only four reported cases of IT-CAT studies (Chen et al, 2020; Joo et al, 2020; Stark & Chernyshenko, 2011; Stark et al, 2012). Such studies are few mainly because this research area is quite new and ipsative tests have several salient features that raise great challenges to researchers (as discussed below).…”

“…With respect to item selection algorithms, Stark and his colleagues used the item information criterion (e.g., 90% of the maximum information), whereas Chen et al (2020) and Joo et al (2020) used the A-optimality criterion as an item selection criterion. The results showed these criteria were effective in general.…”

Detecting DIF in Multidimensional Forced Choice Measures Using the Thurstonian Item Response Theory Model

“…This approach is referred to as a direct estimation strategy. GGUM-RANK information functions and indices were developed (Joo et al, 2018) and used most recently to examine the benefits of MFC triplet computerized adaptive testing (CAT; Joo, Lee, & Stark, 2020).…”

“…Each dimension was measured by 12 statements, and a negatively keyed statement was included in each item. We selected generating parameters and intercorrelations by benchmarking previous MFC simulations (e.g., Brown & Maydeu-Olivares, 2011, 2012Bürkner et al, 2019;Joo et al, 2020). Loadings (lÞ were randomly sampled from U(0.8, 1.3) and thresholds (gÞ were randomly sampled from U(À1, 1).…”