Sample Size Requirements for Applying Mixed Polytomous Item Response Models: Results of a Monte Carlo Simulation Study

Abstract

Mixture models of item response theory (IRT) can be used to detect inappropriate category use. Data collected by panel surveys where attitudes and traits are typically assessed by short scales with many response categories are prone to response styles indicating inappropriate category use. However, the application of mixed IRT models to this data type can be challenging because of many threshold parameters within items. Up to now, there is very limited knowledge about the sample size required for an appropriate performance of estimation methods as well as goodness-of-fit criteria of mixed IRT models in this case. The present Monte Carlo simulation study examined these issues for two mixed IRT models [the restricted mixed generalized partial credit model (rmGPCM) and the mixed partial credit model (mPCM)]. The population parameters of the simulation study were taken from a real application to survey data which is challenging (a 5-item scale with an 11-point rating scale, and three latent classes). Additional data conditions (e.g., long tests, a reduced number of response categories, and a simple latent mixture) were included in this simulation study to improve the generalizability of the results. Under this challenging data condition, for each model, data were generated based on varying sample sizes (from 500 to 5,000 observations with a 500-step). For the additional conditions, only three sample sizes (consisting of 1,000, 2,500, and 4,500 observations) were examined. The effect of sample size on estimation problems and accuracy of parameter and standard error estimates were evaluated. Results show that the two mixed IRT models require at least 2,500 observations to provide accurate parameter and standard error estimates under the challenging data condition. The rmGPCM produces more estimation problems than the more parsimonious mPCM, mostly because of the sparse tables arising due to many response categories. These models exhibit similar trends of estimation accuracy across sample sizes. Under the additional conditions, no estimation problems are observed. Both models perform well with a smaller sample size when long tests were used or a true latent mixture includes two classes. For model selection, the AIC3 and the SABIC are the most reliable information criteria.

The Effect of Rating Scale Length on the Occurrence of Inappropriate Category Use for the Assessment of Job Satisfaction: an Experimental Online Study

“…Forty-three individuals did not provide any valid values on the items of job satisfaction and were excluded from the analysis sample. According to current simulation findings (e.g., Cho 2013;Huang 2016;Jin and Wang 2014;Kutscher et al 2019), this sample size (namely 2000-2500 individuals per condition) should be sufficient for the application of the mixed polytomous IRT model to display optimal performance. Women comprised more than half of the entire sample (61%).…”

“…Within each experimental condition, we estimated the multidimensional rmGPCM including up to five latent classes and determined the best-fitting solution using the Bayesian information criterion (BIC; Schwarz 1978), which works well and is consistent in the context of complex models and large sample sizes (Dziak et al 2012). We purposely chose neither the Akaike's information criterion with the tripled number of model parameters (AIC3; Bozdogan 1994) nor the sample-size adjusted BIC (SABIC; Sclove 1987), both of which showed a good performance for model selection in unidimensional polytomous IRT models (Kutscher et al 2019). However, there is a lack of evidence concerning their performance for multidimensional IRT models.…”

Dimensionality and psychometric analysis of DLQI in a Brazilian population

“…The sample was a result of pooled data from 14 crosssectional studies in Brazil that utilized the DLQI-BRA to assess HRQOL [25][26][27][28][29][30][31][32][33]. The sample size (n = 1286) was assumed to be sufficient in the IRT parameter estimation, dimensionality assessment and DIF and generalized linear model analyses, adjusted for up to eight dummy variables [50][51][52]. All data was related to completely filled questionnaires; there was no available information regarding the number of distributed questionnaires or the percentage of incomplete questionnaires in each original study.…”

A Study on the Identification of Latent Classes Using Mixture Item Response Theory Models: TIMSS 2015 Case

“…The analysis output revealed that the item parameters estimated for Class 1 were labeled as Class 2. This problem was solved by taking the estimated item parameter values as starting values (Kutscher et al, 2019). When item difficulty parameters are examined for South Africa data, it is observed that item difficulty parameters in Class 1 range from -3.017 (item 3) to 2.847 (item 1) while item difficulty parameters in Class 2 range from -0.639 (item 15) to 5.987 (item 1).…”

“…In this case, the label switching problem can be solved by taking the estimated item parameter values as starting values. Model-data fit index values are not affected by Label Switching (Kutscher, Eid, & Crayen, 2019). Table 2 shows the information criteria indices obtained from the analyses aimed at determining which Mixture IRT model (Rasch, 1PL, 2PL, and 3 PL) the data obtained from the science subtest for Singapore, Turkey, and South Africa that attended the 8th grade TIMSS 2015 fits best: When AIC and BIC values are examined in general, it is observed that the AIC and BIC values for Singapore data are lower in the two-class Mixture Rasch model.…”