Assessing the validity of an IAU General English Achievement Test through hybridizing differential item functioning and differential distractor functioning

Abstract

The current study sought to examine the validity of a General English Achievement Test (GEAT), administered to university students in the fall semester of 2018–2019 academic year, by hybridizing differential information (DIF) and differential distractor function (DDF) analytical models. Using a purposive sampling method, from the target population of undergraduate students studying in different disciplines at Islamic Azad University (IAU), (Isfahan branch), a sample of 835 students taking GEAT were selected. The 60-item multiple-choice test comprised four sub-sections; namely, vocabulary, grammar, cloze test, and reading comprehension. The students’ test scores served as the targeted data and the validity of the test was examined through the application of Cochran-Mantel-Haenszel (CMH) and multinomial log-linear regression models for detecting DIF and DDF, respectively. To account for the assumption of uni-dimensionality, the test sub-sections were analyzed independently. Furthermore, the assumption of local independence was checked based on correlational analysis and no extreme values were observed. The results of the study identified five moderate-level DIF items and one DDF item signaling an adverse effect on test fairness due to the existing biased items. Notably, these findings may have important implications for both language policymakers and test developers.

The use of Virtual Learning Environment for students effective engagement in the Higher Education Institutions through knowledge management and blended learning'

“…The advancing phase of data analysis provided further facts to differentiate between the students based on their chosen five set of factors to better understand their engagement through blended learning. The data analysis outcomes were evaluated applying the Differential Item Functioning (DIF) analysis, which indicated that students' various subgroups responded in a different way to the elevated substances (Jamalzadeh et al, 2021). These findings further supported the detection of unique commitment of student at various substances from high to low levels.…”

Psychometric Properties of an Instrument to Evaluate Students’ Perception of Learning Objects in Statistics

“…Participants were selected by intentional sampling according to the purpose of the study. The intentional sampling method is considered a non-probabilistic sampling whose main purpose is to produce a sample that can be assumed to be logically representative of the population (Bhardwaj, 2019; Jamalzadeh et al, 2021).…”

Investigating the Distractors to Explain DIF Effects Across Gender in Large-Scale Tests With Non-Linear Logistic Regression Models

“…Many types of research have been carried out to determine the validity and reliability of large-scale assessments because the performance of examinees on these tests has a critical impact on their educational admissions and future careers. For that reason, validity is a core feature of any kind of assessment assumed to be accurate and fair (Bond et al, 2003;Jamalzadeh et al, 2021). Therefore, the goal of test developers and testing companies is to increase the validity and reliability of tests by decreasing any types of confounding factors and errors to ensure fairness across different subgroups.…”

“…DIF analyses, in general, are integrated with DDF analyses to explain the DIF effects. The relationship between DIF and DDF effects is casual rather than correlational (Deng, 2020;Jamalzadeh et al, 2021). Penfield (2010) found that the DDF effect may cause uniform DIF and partially explained by DDF, whereas nonuniform DIF may indicate the variation of signs within distractors.…”