Issues Affecting Item Response Theory Fit in Language Assessment: A Study of Differential Item Functioning in the Iranian National University Entrance Exam

Abstract

This study aimed at examining the issues affecting the use of IRT models in investigating differential item functioning in high stakes testing. It specifically focused on the Iranian National University Entrance Exam (INUEE) Special English Subtest. A sample of 200,000 participants was randomly selected from the candidates taking part in the INUEE 2003 and 2004 respectively. The data collected in six domains of vocabulary, grammar, word order, language function, cloze test and reading comprehension were analyzed to evaluate the applicability of item response theory (IRT; Embretson & Reise, 2000), including the use of IRT for assessing differential item functioning (DIF; Zumbo, 2007). Substantial model-data misfit was observed in calibrations using PARSCALE and BILOG MG software (Scientific Software International, 2004). Additional analysis through Xcalibre and Iteman 4 (Assessment Systems Corporation, 2010) suggested that item response theory, including IRT-based DIF analysis, is not applicable when the test administered is noticeably beyond the participants’ level of capability, when the test is speeded, or if students are penalized for their wrong answers

Detecting Item Bias in an Anatomy & Physiology Test for Nursing Students using Item Response Theory

“…Before the interpretation of the DIF analysis is made, items that have a fit issue on the model and are beyond the ability of the examinees should be noted as such items may cause IRT cannot be applied to investigate DIF (Ahmadi & Thompson, 2012). Therefore, item with fit issue and beyond the examinees' ability should be excluded from the DIF analysis.…”

“…This study has applied IRT-based software known as Xcalibre in calibrating research dichotomous data. Ahmadi and Thompson (2012) remind, it should be noted that fit issues on the IRT model will cause IRT to not apply for DIF investigation. In fact, the analysis using Xcalibre and Iteman 4 software including others IRT-based DIF analysis cannot be applied when the administered test is found to be beyond the ability of the examinee, if the test is in the form of speeded or if the examinee is penalized for the wrong response (Ahmadi & Thompson, 2012).…”