A Bifactor Multidimensional Item Response Theory Model for Differential Item Functioning Analysis on Testlet-Based Items

Abstract

A differential item functioning (DIF) detection method for testlet-based data was proposed and evaluated in this study. The proposed DIF model is an extension of a bifactor multidimensional item response theory (MIRT) model for testlets. Unlike traditional item response theory (IRT) DIF models, the proposed model takes testlet effects into account, thus estimating DIF magnitude appropriately when a test is composed of testlets. A fully Bayesian estimation method was adopted for parameter estimation. The recovery of parameters was evaluated for the proposed DIF model. Simulation results revealed that the proposed bifactor MIRT DIF model produced better estimates of DIF magnitude and higher DIF detection rates than the traditional IRT DIF model for all simulation conditions. A real data analysis was also conducted by applying the proposed DIF model to a statewide reading assessment data set.

Detecting Multidimensional Differential Item Functioning with the Multiple Indicators Multiple Causes Model, the Item Response Theory Likelihood Ratio Test, and Logistic Regression

“…The effect sizes indicated that DIF type, DIF magnitude, and sample size were highly influential on rejection rates, whereas test length did not seem to have a strong impact on rejection rates. These results of MANOVA were consistent with the patterns of the rejection rates presented in Tables 2 and 3. dIsCUssIoN There have been several methods proposed for the detection of DIF in multidimensional item response data (e.g., Stout et al, 1997;Mazor et al, 1998;Fukuhara and Kamata, 2011;Suh and Cho, 2014;Lee et al, 2016). Among these methods, the logistic regression, the MIMIC-interaction model, and the IRT-LR test are the most readily available for detecting DIF in dichotomously and polytomously scored items because of their ease of use.…”

The Effects of Testlets on Reliability and Differential Item Functioning

“…Moreover, on reviewing the DIF studies available in literature, it was concluded that the DIF was found to be trivial at the item level with analyses without considering the testlet effects and became clear with analyses considering the testlet effects (Fukuhara, 2009;Fukuhara & Kamata, 2011;Sedivy, 2009;Wainer et al, 1991). This does not lead to serious differences when there is large enough data sample ; but as the sample and the DIF becomes smaller, the methods considering the testlet effects displays better performance.…”

“…In other words, through this method, the differential testlet functioning (DTF) is derived. Therefore, specific items which lead to DIF cannot be determined (Fukuhara & Kamata, 2011). Since constructing a testlet is demanding, time-consuming work, taking the testlet displaying DTF out of the item pool would not be a desirable case.…”

“…The model suggested by Fukuhara and Kamata (2011) and used in this study is basically the developed form of the bi-factor multidimensional item response theory model for testlets (bi-factor MIRT) suggested by Li et al (2006) and DeMars (2006), and can be represented mathematically as in Equation 2: In P(y ij = 1)…”

“…Since the aim was to determine the effects of tests composed of testlets on DIF, two methods, one of which considered testlet effects and one of which did not consider testlet effects, were used in this study. The methods used were recommended by Fukuhara and Kamata (2011), and were based on IRT. In determining the DIF levels in this study, the measure of 0.426 was considered to be significant at B level whereas the measure of 0.638 was considered to be significant at C level for both methods (Vaughn, 2006).…”