The application of Kunnan’s test fairness framework (TFF) on a reading comprehension test

Abstract

The present study tries to investigate the fairness of an English reading comprehension test employing Kunnan's (2004) test fairness framework (TFF) as the most comprehensive model available for test fairness. The participants of this study comprised 300 freshman students taking general English course chosen based on the availability sampling, three test developers, and seven university officials, administering the test. The main instrument is the teacher-made reading comprehension test to examine its validity, reliability, and differential item functioning (DIF). Furthermore, to examine the other modules of TFF, namely, access, administration, and social consequence, a questionnaire and semi-structured interviews with the test developers and test administrators were applied. In order to analyze the data, exploratory factor analysis is used to evaluate test validity employing Minitab software. Moreover, t test and ANOVA were used to examine the disparate impact of the test using SPSS package. Furthermore, the Mantel-Haenszel procedure is applied to determine the DIF, while coding the required formulas in R programming. The frequencies of different aspects of access and test administration are explained and consolidated by qualitative data gleaned from interview sessions. Examining the data with respect to TFF modules, it was concluded the test must be enhanced in terms of validity, while it is totally fair. The statistical procedure and the mixed research design implemented in the present study can be a sound model to be applied by test developers to enhance the test fairness of the exams.

Examining the subjective fairness of at-home and online tests: Taking Duolingo English Test as an example

“…Kunnan’s [ 27 ] TFF, prioritizing the role of test fairness, has broadened the span of fairness. The framework has been adopted in the research that investigates the fairness of large-scale assessments (e.g., [ 29 , 30 ]) and small-scale assessments (e.g., [ 10 , 12 ]). For large-scale assessments, Hamid et al [ 29 ] examined the fairness and justice of the IELTS and found a large proportion of participants believed that the test was not an accurate measure of their language abilities, and they were not satisfied with the IELTS justness in that they were not sure whether the test was a vehicle for increasing income or its suitableness for immigration purposes.…”

“…For large-scale assessments, Hamid et al [ 29 ] examined the fairness and justice of the IELTS and found a large proportion of participants believed that the test was not an accurate measure of their language abilities, and they were not satisfied with the IELTS justness in that they were not sure whether the test was a vehicle for increasing income or its suitableness for immigration purposes. Moghadam and Nasirzadeh [ 30 ] investigated the fairness of a locally developed English proficiency test and found that the test was acceptably fair from the perspectives of access, administration, and social consequences. But the validity of the test should be enhanced; and one item was flagged as gender DIF which needed further revision.…”

Assessing the validity of an IAU General English Achievement Test through hybridizing differential item functioning and differential distractor functioning

“…Clearly, the consequential matters of test fairness and equity are quintessentially important because all examinees should enjoy equal opportunities to perform satisfactorily on a large-scale assessment, and hence being treated equitably in terms of their test scores (Moghadam & Nasirzadeh, 2020). The distinction between item bias and item impact is defined and clarified by the purpose of the measure.…”