Likelihood-Based Item-Fit Indices for Dichotomous Item Response Theory Models

Abstract

New goodness-of-fit indices are introduced for dichotomous item response theory (IRT) models. These indices are based on the likelihoods of number-correct scores derived from the IRT model, and they provide a direct comparison of the modeled and observed frequencies for correct and incorrect responses for each number-correct score. The behavior of Pearson's χ 2 (S-X 2 ) and the likelihood ratio G 2 (S-G 2 ) was assessed in a simulation study and compared with two fit indices similar to those currently in use (Q 1 -X 2 and Q 1 -G 2 ). The simulations included three conditions in which the simulating and fitting models were identical and three conditions involving model misspecification. S-X 2 performed well, with Type I error rates close to the expected .05 and .01 levels. Performance of this index improved with increased test length. S-G 2 tended to reject the null hypothesis too often, as did Q 1 -X 2 and Q 1 -G 2 . The power of S-X 2 appeared to be similar for all test lengths, but varied depending on the type of model misspecification. Index terms: chi-square distribution, dichotomous items, goodness-of-fit, item fit, item response theory (item fit), likelihood ratio statistic, Pearson statistic.Item response theory (IRT) is a collection of modeling techniques for the analysis of items, tests, and persons. An IRT model for dichotomous item responses generally specifies that the probability of response pattern x iswhere x is the response vector, T i (θ) is the probability of a correct response on item i as a function of the trait θ, and φ(θ) is the population distribution for θ. The three-parameter logistic model (3PLM),where a i is the slope parameter, b i is the location parameter, and c i is the lower asymptote parameter, is commonly used for multiple-choice items (see Lord, 1980). In some situations, the twoparameter logistic model (2PLM) is used. This model is equivalent to the 3PLM with c i = 0. The one-parameter logistic model (1PLM) additionally restricts a i = a for all items. IRT applications are based on estimating model parameters (a i , b i , and c i ). These estimates are usually obtained by maximizing the likelihood50

How many IRT parameters does it take to model psychopathology items?

“…For example, BILOG (Mislevy & Bock, 1998) provides item chi-square fit statistics, but these are problematic on a number of grounds and cannot be used to compare the fit of different models on an item-by-item basis. See Orlando and Thissen (2000) for further comment. where we take the sum of squared differences (x ij ) and predicted (P ij ) item responses for every cell in the Persons (J) × Items (I) matrix.…”

Statistical Considerations in the Psychometric Validation of Outcome Measures

“…Differential item functioning is detected by comparing the item responses of people who have the same underlying true value of the construct. Because this true value is not known, it is estimated using different methods, for example, the summed score across all items, or an estimate based on item response theory [31,32,34,[44][45][46]. The analyses can be carried out in two stages: initially with all items and then without items exhibiting bias [10,16].…”

The Correlation Between Item Parameters and Item Fit Statistics

“…The χ 2 G statistic was included in the simulations, as in Dodeen (2004), as well as the S − χ 2 and S − G 2 statistics suggested by Orlando and Thissen (2000). For computing S − χ 2 and S − G 2 , the examinees were divided into G groups based on their raw scores.…”

“…Item fit is a major area of interest in model checking. Though researchers have suggested several different item fit statistics (e.g., Bock, 1972;Orlando & Thissen, 2000;Sinharay, 2006;Stone, 2000;Stone & Zhang, 2003;Glas & Suarez-Falcon, 2003;Yen, 1981), there is a lack of sufficient knowledge regarding factors that usually cause item misfit. For example, more appropriate assessments have resulted because the substantial existing knowledge of factors affecting differential item functioning, or DIF, (see, for example, Schmitt, Holland, & Dorans, 1993, and the references therein) often help test developers to control the number of items with DIF.…”

“…Several item fit statistics have been suggested recently, by researchers such as Glas and Suarez-Falcon (2003), Orlando and Thissen (2000), Sinharay (2006), Stone (2000), and Stone and Zhang (2003). There is a need to study the same relationships as studied by Dodeen (2004), but with these recently developed item fit statistics; if the relationships hold for these newer statistics as well, there will be sufficient reason to be careful about test construction.…”