An Assessment of the Nonparametric Approach for Evaluating the Fit of Item Response Models

Abstract

As item response theory has been more widely applied, investigating the fit of a parametric model becomes an important part of the measurement process. There is a lack of promising solutions to the detection of model misfit in IRT. Douglas and Cohen introduced a general nonparametric approach, RISE (Root Integrated Squared Error), for detecting model misfit. The purposes of this study were to extend the use of RISE to more general and comprehensive applications by manipulating a variety of factors (e.g., test length, sample size, IRT models, ability distribution). The results from the simulation study demonstrated that RISE outperformed G 2 and S-X 2 in that it controlled Type I error rates and provided adequate power under the studied conditions. In the empirical study, RISE detected reasonable numbers of misfitting items compared to G 2 and S-X 2 , and RISE gave a much clearer picture of the location and magnitude of misfit for each misfitting item. In addition, there was no practical consequence to classification before and after replacement of misfitting items detected by three fit statistics.

Using Bayesian Nonparametric Item Response Function Estimation to Check Parametric Model Fit

“…In contrast, nonparametric methods (e.g., Ramsay, 1991) estimate the IRF without assuming any of its mathematical form and can adapt to irregularities in the data. Those nonparametric IRF estimation methods are often used to check the model fit of the parametric models by assessing the functional shape of departures from the parametric models (Junker & Sijtsma, 2001; Lee et al, 2009; Liang et al, 2014; Liang & Wells, 2009; Sijtsma & Junker, 2006; Stout, 2001; Wells & Bolt, 2008). An advantages of nonparametric model fit methods, compared with chi-square-based model fit methods (McKinley & Mills, 1985; Orlando & Thissen, 2000; Yen, 1981), are that nonparametric model fit methods can provide graphical representation of the misfit (Liang & Wells, 2009).…”

“…All model fit methods need a distribution to determine the significance level of the fit statistics used to judge the fit of the item. In studies applying nonparametric model fit methods, the bootstrapping procedure was performed to construct an empirical distribution of fitting statistics (Douglas & Cohen, 2001; Lee et al, 2009; Liang et al, 2014; Liang & Wells, 2009; Wells & Bolt, 2008). Previous used bootstrapping procedure did not consider the uncertainty arising from parameter estimation because it used previously obtained item parameter estimates.…”

A Bias-Corrected RMSD Item Fit Statistic: An Evaluation and Comparison to Alternatives

“…However, there exists no theoretical basis regarding the distribution of the residuals under the null hypothesis of perfect model fit and hence no statistic for testing the null hypothesis. For most statistics, studies found inflated Type I error rates, especially for large sample sizes, as well as a lack of power to detect item misfit (Chon, Lee, & Ansley, 2013; DeMars, 2005; Glas & Suárez Falcón, 2003; Liang, Wells, & Hambleton, 2014; Orlando & Thissen, 2000; Stone & Zhang, 2003). Note that the statistics mentioned so far and the statistics investigated in the remaining article focus on detecting misfit with regard to the functional form assumed by the parametric model.…”

Karakteristik Butir Soal: Classical Test Theory vs Item Response Theory?

“…Model ini selanjutnya dikenal tes modern atau tes respon butir. Menurut teori respon butir, perilaku seseorang dapat dijelaskan oleh karakteristik orang yang bersangkutan sampai pada batasbatas tertentu(Mardapi, 2012) Van der Linden & Hambleton (2013) menyatakan bahwa teori respon butir (IRT) merupakan salah satu cara untuk menilai kelayakan butir dengan membandingkan rerata penampilan butir terhadap tampilan bukti kemampuan kelompok yang diramalkan oleh model (Liang, Wells, & Hambleton, 2014). mengatakan bahwa "Item response theory (IRT) is a powerful scaling technique with appealing features such as the invariance of item and ability parameter values".…”