Applying the Mantel‐Haenszel Procedure to Complex Samples of Items

Allen, Nancy L.; Donoghue, John R.

doi:10.1111/j.1745-3984.1996.tb00491.x

Cited by 22 publications

(47 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These results are consis tent with the findings of Allen and Donoghue (1994), who have shown in certain cases extremely large Type I errors (as high as 0.8 for a nominal level of 0.05) for simulation models designed to realistically model the American NAEP assessment examination.…”

Section: Extension Of Sibtest To Ordered Polytomous Itemssupporting

confidence: 88%

Differential Item Functioning: its Multidimensional Model and Resulting Sibtest Detection Procedure

Bolt

Stout

1996

Behaviormetrika

View full text Add to dashboard Cite

A multidimensional model of differential item functioning (DIF) developed by Shealy and Stout is presented.It explains how individual items combine to produce differential test functioning (DTF) at the test level.Recent developments based on this approach, including development of the DIF/DTF detection procedure SIBTEST, are surveyed. The Shealy-Stout model not only offers insight into how DIF can occur, but suggests methods for investigating the root causes of DIF, which are useful to both substantive and statistical researchers of DIF.This new modeling paradigm offers the possibility of the proactive reduction of DIF in tests at the item manufacturing stage.SIBTEST is a nonparametric procedure that both tests for and estimates the amount of DIF in an item or set of items while controlling inflated Type I error by using a regression correction technique.It is shown to perform as well as and in many realistic situations better than other popular DIF assessment procedures. Recent modifications to the procedure demonstrate that it can be an effective tool for examining both crossing DIF and, through the use of kernel smoothing, local DIF. The procedure has been extended for use with tests containing polytomous items as well as tests that are intentionally multidimensional.Many real and simulated data analyses are presented.

show abstract

Section: Extension Of Sibtest To Ordered Polytomous Itemssupporting

confidence: 88%

Differential Item Functioning: its Multidimensional Model and Resulting Sibtest Detection Procedure

Bolt

Stout

1996

Behaviormetrika

View full text Add to dashboard Cite

show abstract

“…Similar assumptions have been made in the Allen and Donoghue (1996) study. Because Block A is present in all three booklets, it is said that these booklets are ''anchored'' by a common-item Block A (see Table 2).…”

Section: New Methodsmentioning

confidence: 67%

Equated Pooled Booklet Method in DIF Testing

Cheng

Chen

Qian

et al. 2013

Applied Psychological Measurement

View full text Add to dashboard Cite

Differential item functioning (DIF) analysis is an important step in the data analysis of large-scale testing programs. Nowadays, many such programs endorse matrix sampling designs to reduce the load on examinees, such as the balanced incomplete block (BIB) design. These designs pose challenges to the traditional DIF analysis methods. For example, as difficulty levels often vary across booklets, examinees with same booklet scores may be disparate in ability. Consequently, DIF procedures based on matching total scores at the booklet level may cause misplacement of examinees and inflation in measurement errors. Therefore, modification to traditional DIF procedures to better accommodate the BIB design becomes important. This article introduces modification of current simultaneous item bias test (SIBTEST) procedure for the DIF analysis method when multiple booklets are used. More specifically, examinees will be pooled across booklets, and the matching will be based on transformed booklet scores after common block equating/linking. Simulations are conducted to compare the performance of this new method, the equated pooled booklet method against that of the current pooled booklet method, in terms of both Type I error control and power. Four factors are considered in the simulation-the DIF effect size, item difficulty, impact, and the length of common block. Results show that the equated pooled booklet method in general improves power while keeping Type I error under control. The advantage of the new method is the most pronounced when the traditional method struggles, for example, when the item is difficult or there is impact.

show abstract

“…When tests are long or when the variation of discrimination across test items is not large, these procedures can be effective (see for example Shealy & Stout, 1993a, or Zwick et al). However for short or moderate length tests and large but realistic levels of variation in item discrimination, the Type I error rate can become substantially larger than the nominal level (see Allen & Donoghue, 1996;Chang, Mazzeo, & Roussos, 1996; for simulated DIF studies using IRT models designed by the authors to model actual test settings. Including the studied item in the matching score X does nothing to prevent the stochastic ordering of the distributions of 9 R | X R = x and 9 F | X F = x.…”

Section: Notes On the Mantel-haenszelmentioning

confidence: 99%

Improved Type I Error Control and Reduced Estimation Bias for DIF Detection Using SIBTEST

Jiang

Stout

1998

Journal of Educational and Behavioral Statistics

View full text Add to dashboard Cite

One emphasis in the development and evaluation of SIBTEST has been the control of Type I error (false flagging of non-differential item functioning [DIF] items) inflation and estimation bias. SIBTEST has performed well in comparative simulation studies of Type I error and estimation bias relative to other procedures such as the Mantel-Haenszel and Logistic Regression. Nevertheless it has for a minority of cases that might occur in applications displayed sizable Type I error inflation and estimation bias. A vital part of SIBTEST is the regression correction, which adjusts for the Type I error-inflating and estimation-biasing influence of group target ability differences by using the linear regression of true on observed matching subtest scores from Classical Test Theory In this paper, we propose a new regression correction, using essentially a two-segment piecewise linear regression of the true on observed matching subtest scores. A realistic simulation study of the new approach shows that when there is a clear group ability distributional difference, the new approach displays improved SIBTEST Type I error performance; when there is no group ability distributional difference, its Type I error rate is comparable to the current SIBTEST. We have also conducted a power study which indicates that the new approach has on average similar power as the current SIBTEST. We concluded that the new version of SIBTEST, although not perfectly robust, seems appropriately robust against sizable Type I error inflation, while retaining other desirable features of the current version.Shealy and Stout (1993a) have developed a statistical test, SIBTEST, to detect and estimate differential item functioning (DIF) resulting from either a single item or a bundle of items. For a single studied item, DIF occurs when the marginal (i.e., secondary abilities have been integrated out) item response functions (IRFs) for different demographic populations are not identical with respect

show abstract

Applying the Mantel‐Haenszel Procedure to Complex Samples of Items

Cited by 22 publications

References 14 publications

Differential Item Functioning: its Multidimensional Model and Resulting Sibtest Detection Procedure

Differential Item Functioning: its Multidimensional Model and Resulting Sibtest Detection Procedure

Equated Pooled Booklet Method in DIF Testing

Improved Type I Error Control and Reduced Estimation Bias for DIF Detection Using SIBTEST

Contact Info

Product

Resources

About