The Performance of IRT Model Selection Methods With Mixed-Format Tests

Abstract

When tests consist of multiple-choice and constructed-response items, researchers are confronted with the question of which item response theory (IRT) model combination will appropriately represent the data collected from these mixed-format tests. This simulation study examined the performance of six model selection criteria, including the likelihood ratio test, Akaike's information criterion (AIC), corrected AIC, Bayesian information criterion, Hannon and Quinn's information criterion, and consistent AIC, with respect to correct model selection among a set of three competing mixed-format IRT models (i.e., one-parameter logistic/partial credit [1PL/PC], two-parameter logistic/generalized partial credit [2PL/GPC], and three-parameter logistic/generalized partial credit [3PL/GPC]). The criteria were able to correctly select less parameterized IRT models, including the PC, 1PL, and 1PL/PC models. In contrast, the criteria were less able to correctly select more parameterized IRT models, including the GPC, 3PL, and 3PL/GPC models. Implications of the findings and recommendations are discussed.

The Impact of Varied Discrimination Parameters on Mixed-Format Item Response Theory Model Selection

“…A simulation study was conducted that duplicated the Whittaker et al (2012) study with the exception of varied discrimination parameters to determine if low discrimination parameter variance was the factor that contributed to the less than satisfactory performance of the model selection criteria when selecting the more parameterized, generating IRT models. Conditions that were varied, as in the original study, included sample size, proportion of dichotomous and polytomous items on the test, total score points, and generating mixed-format IRT model, which are briefly discussed subsequently (see Whittaker et al, 2012, for more detailed information concerning the manipulated conditions in their study).…”

“…More homogeneous discrimination parameters would lead to the selection of the most parsimonious models in which the discrimination parameter is constant (i.e., the PC, 1PL, and 1PL/PC models). This pattern was demonstrated by the Bayesian information criterion (BIC; Schwarz, 1978) and Bozdogan's (1987) consistent AIC (CAIC) in Whittaker et al (2012). The tendency of the remaining model selection criteria (viz., LRT, AIC, AICC, and HQIC) to have incorrectly selected the 2PL and 2PL/GPC models as compared with the 3PL and 3PL/GPC models, respectively, may have also been because of homogenous discrimination parameters.…”

“…Total Score Points on Test Whittaker et al (2012) manipulated the total score points on a test instead of manipulating test length directly. The total number of scores points attributed to dichotomously scored and/or polytomously scored items was either 26 or 50 points.…”

“…Employing both 26 and 50 total score points on a test with each of the five proportions of item type combinations resulted in 10 distinctive tests with varying lengths. Whittaker et al (2012) used three mixed-format IRT models as generating models, including the 1PL/PC, 2PL/GPC, and 3PL/GPC models. The 1PL/PC model was not employed as a generating model in the current study because it does not consist of a discrimination parameter and was accurately selected in the Whittaker et al (2012) study.…”

“…More specifically, the less parameterized 2PL model may have fit better, as assessed by the model selection criteria, than the more parameterized 3PL model because there was variability in only two of the three parameters estimated in the 3PL model, including the guessing parameter and the difficulty parameters. Thus, the purpose of this article is to examine the performance of the model selection criteria used in Whittaker et al (2012) when selecting among mixed-format IRT models with more variable discrimination parameters.…”