Gender-related Differential Item Functioning Analysis on an ESL Test

Abstract

Differential item functioning (DIF) is a technique used to examine whether items function differently across different groups. The DIF analysis helps detect bias in an assessment to ensure the fairness of the assessment. However, most of the previous research has focused on high-stakes assessments. There is a dearth in research that laying emphasis on low-stakes assessments, which is also significant for the test development and validation process. Additionally, gender difference in test performance is always a particular concern for researchers to evaluate whether a test is fair or not. This present study investigated whether test items of the General English Proficiency Test for Kids (GEPT-Kids) are free of bias in terms of gender differences. A mixed-method sequential explanatory research design was adopted with two phases. In phase I, test performance data of 492 participants from five Chinese speaking cities were analyzed by the Mantel-Haenszel (MH) method to detect gender DIF. In phase II, items that manifested DIF were subject to content analysis through three experienced reviewers to identify possible sources of DIF. The results showed that three items were detected with moderate gender DIF through statistical methods and three items were identified as possible biased items by expert judgment. The results provide preliminary contributions to DIF analysis for low-stakes assessment in the field of language assessment. Besides, young language learners, especially in the Chinese context, have been drawn renewed attention. Thus, the results may also add to the body of literature that can shed some light on the test development for young language learners.

Modelling the lexical complexity of homogenous texts: a time series approach

“…More examples illustrate how homogeneity plays an essential role in the theoretical formulation or experiment design. Such instances include the cross corpora analysis (Kilgarriff, 2001 ; Kilgarriff and Grefenstette, 2003 ; Denoual, 2005 ; Smith and Kelly, 2002 ), topical dispersion analysis (Sahlgren and Karlgren, 2005 ), second language (L2) writing classification (Crossley and McNamara, 2011 ), and language assessment validations (Yao and Chen, 2020 ; Jang et al 2015 ).…”

Grade-Related Differential Item Functioning in General English Proficiency Test-Kids Listening

“…Similarly, Zumbo (2007) also pointed out this problem and called for exploring the reasons for DIF. In fact, some studies have made efforts to answer this why question (e.g., Li et al, 2004 ; Yao and Chen, 2020 ). However, only a weak relationship was found between gender DIF and task types, but no convincing explanations could be provided (see detailed discussion in the next section).…”