Identifying Problematic Item Characteristics With Small Samples Using Mokken Scale Analysis

Abstract

Researchers frequently use Mokken scale analysis (MSA), which is a nonparametric approach to item response theory, when they have relatively small samples of examinees. Researchers have provided some guidance regarding the minimum sample size for applications of MSA under various conditions. However, these studies have not focused on item-level measurement problems, such as violations of monotonicity or invariant item ordering (IIO). Moreover, these studies have focused on problems that occur for a complete sample of examinees. The current study uses a simulation study to consider the sensitivity of MSA item analysis procedures to problematic item characteristics that occur within limited ranges of the latent variable. Results generally support the use of MSA with small samples ( N around 100 examinees) as long as multiple indicators of item quality are considered.

Theoretical and Practical Foundations of Mokken Scale Analysis in Psychology

“…Second, it offers a more flexible framework for applications where parametric models fit the data poorly, which allows for more flexible scales than are possible with parametric models. Finally, MSA procedures make it possible to use a smaller number of items and samples (Wind, 2022) than those used in large-scale tests adjusted with parametric models, given that, instead of trying to estimate parameters, MSA tests the assumptions of the models.…”

Validity evidence for the coping strategy indicator-short version (CSI-S) among psychology students

“…The study anticipated a relatively small sample size, but the sampling process aimed to ensure the voluntary participation of examinees. This anticipation also involved using nonparametric modeling (Mokken scaling analysis; see below), in which context at least 100 participants represented a reasonable sample size 48 . One author managed the academic coordination of the institutions with the goal of obtaining permission to administer the instruments during class time.…”

“…Methodologically, an ancillary contribution of the study pertains to the use of nonparametric item response theory methodology to maximize the internal validity of the study in a challenging situation in terms of sample size and variability. More precisely, due to the small size of the study and the exploratory objective of the participant group, Mokken scaling analysis (MSA) may be a reasonable choice 48 , 49 .…”