How Much g Is in the Distractor? Re-Thinking Item-Analysis of Multiple-Choice Items

Abstract

Distractors might display discriminatory power with respect to the construct of interest (e.g., intelligence), which was shown in recent applications of nested logit models to the short-form of Raven’s progressive matrices and other reasoning tests. In this vein, a simulation study was carried out to examine two effect size measures (i.e., a variant of Cohen’s ω and the canonical correlation RCC) for their potential to detect distractors with ability-related discriminatory power. The simulation design was adopted to item selection scenarios relying on rather small sample sizes (e.g., N = 100 or N = 200). Both suggested effect size measures (Cohen’s ω only when based on two ability groups) yielded acceptable to conservative type-I-error rates, whereas, the canonical correlation outperformed Cohen’s ω in terms of empirical power. The simulation results further suggest that an effect size threshold of 0.30 is more appropriate as compared to more lenient (0.10) or stricter thresholds (0.50). The suggested item-analysis procedure is illustrated with an analysis of twelve Raven’s progressive matrices items in a sample of N = 499 participants. Finally, strategies for item selection for cognitive ability tests with the goal of scaling by means of nested logit models are discussed.

Goodman–Kruskal gamma and Dimension-Corrected Gamma in Educational Measurement Settings

“…In educational measurement settings, G and partial G (Goodman & Kruskal, 1954; see also Davis, 1967) are used, to some extent although rarely, in item analysis (e.g., Forthmann et al, 2020;Kreiner & Christensen, 2009;Nielsen et al, 2017;Nielsen & Santiago, 2020). However, remembering that the coefficients G and D are close siblings, they have a strict connection to educational measurement settings because of their connection to rank-biserial correlation coefficient (Cureton, 1956) based on U test statistic (Mann & Whitney, 1947) and rank-polyserial correlation (Metsämuuronen, 2021) based on Jonckheere-Terpstra test statistic (JT; Jonckheere, 1954;Terpstra, 1952); these are special cases of D and G (Newson, 2008;Metsämuuronen, 2021).…”

“…In general, both G and D estimate the probability that two randomly chosen cases have the same order in two variables (γ and δ, respectively; e.g., Van der Ark & Van Aert, 2015;Metsämuuronen, 2021). In measurement modelling settings, this is sometimes interpreted as the relationship between test score and the probability to choose the correct response (Forthmann, et al, 2020based on Love, 1997.…”

Regularized Latent Class Analysis for Polytomous Item Responses: An Application to SPM-LS Data<strong></strong>

“…. , K i denote the item parameters of item i of category k in class c. Such constraints can be replaced by adding ridge-type penalties of the form λ 3 ∑ K i k=0 γ 2 ikc to the fused regularization penalty, where λ 3 is another regularization parameter. By squaring item parameters in the penalty function, they are uniformly shrunk to zero in the estimation.…”

“…[1] composed the last twelve matrices of the Standard Progressive Matrices (SPM-LS) and argued that it could be regarded as valid indicator of general intelligence g. As part of this special issue, the SPM-LS dataset that was analyzed in [1] was reanalyzed in a series of papers applying a wide range of psychometric approaches. In particular, [2] investigated item distractor analysis with a particular focus on reliability, [3] provided additional insights due to dimensionality analysis, [4] applied the Haberman interaction model using the R package dexter, Mokken scaling was employed by [5], and, finally, [6] presented Bayesian item response modeling using the R package brms.…”

Analysis of an Intelligence Dataset

“…Moreover, Bürkner ( 2020 ) later presents how to use his R Bayesian multilevel modeling package ( Bürkner 2017 ) in order to estimate various binary item response theory models, and compares the results with the frequentist approach used in the original paper with the item response theory package ( Chalmers 2012 ). Furthermore, Forthmann et al ( 2020 ) later proposed a new procedure that can be used to detect (or select) items that could present discriminating distractors (i.e., items for which distractor responses could be used to extract additional information). In addition, Partchev ( 2020 ) then discusses issues that relate to the use of distractor information to extract information on ability in multiple choice tests, in particular in the context of cognitive assessment, and presents how to use the R package ( Maris et al 2020 ) to study the binary responses and distractors of the SPM–LS.…”