Assessing Differential Item Functioning in Direct Writing Assessments: Problems and an Example

Abstract

The recent emphasis on various types of performance assessments raises questions concerning the differential effects of such assessments on population subgroups. Procedures for detecting differential item functioning (DIF) in data from performance assessments are available but may be hindered by problems that stem from this mode of assessment. Foremost among these are problems related to finding an appropriate matching variable. These problems are discussed and results are presented for three methods for DIF detection in polytomous items using data from a direct writing assessment. The purpose of the study is to examine the effects of using different combinations of internal and external matching variables. The procedures included a generalized Mantel‐Haenszel statistic, a technique based on meta‐analysis methodology, and logistic discriminant function analysis. In general, the results did not support the use of an external matching criterion and indicated that continued problems may be expected in attempts to assess DIF in performance assessments.

Ensuring the Fairness of Gre Writing Prompts: Assessing Differential Difficulty

“…This is also consistent with previous research (Welch & Miller, 1995). Under such circumstances, one practical suggestion would be to increase the threshold for flagging GRE prompts in the following way:…”

“…In this study, we used two of the methods recommended by Penfield and Lam (2000) as well as the Mantel test, a method that has been widely researched for multiplechoice items. Welch and Miller (1995) studied different DIF detection procedures and different combinations of external and internal matching variables in investigations of DIF for essay items. One of the most important findings from this study was that, in most cases, combining a multiple-choice test score of related content with one essay score as the matching criterion reduced the apparent degree of DIF substantially (when compared to a criterion made up of only scores on the multiple-choice items), and even further when a second essay score was added.…”

“…where T k is the total number of examinees at level k, m jk = A jk + B jk , and Welch & Miller, 1995).…”

Describing and Categorizing Dif in Polytomous Items

“…As in the case of SMD, the matching variable is defined as the sum of the scores on the matching items and the score on the studied item. The procedure has been applied for DIF purposes by several researchers (see Potenza & Dorans, 1995;Zwick & Thayer, 1994, in press;Welch & Miller, 1995).…”

“…At present, ETS has no official policy for screening polytomous items for DIF. Several promising statistical tests for DIF in polytomous items have been investigated (see reviews in Potenza & Dorans, 1995;Zwick & Thayer, 1994, in press;Welch & Miller, 1995), but little work has been conducted on the development of descriptive measures of DIF for these items. This project involved both the evaluation of inferential procedures and the development of DIF indexes and their standard errors.…”