Consequences of Violated Equating Assumptions Under the Equivalent Groups Design

Journal of Educational and Behavioral Statistics

2019

When equating two test forms, the equated scores will be biased if the test groups differ in ability. To adjust for the ability imbalance between nonequivalent groups, a set of common items is often used. When no common items are available, it has been suggested to use covariates correlated with the test scores instead. In this article, we reduce the covariates to a propensity score and equate the test forms with respect to this score. The propensity score is incorporated within the kernel equating framework using poststratification and chained equating. The methods are evaluated using real college admissions test data and through a simulation study. The results show that propensity scores give an increased equating precision in comparison with the equivalent groups design and a smaller mean squared error than by using the covariates directly. Practical implications are also discussed.

Section: Empirical Illustrationmentioning

confidence: 99%

Kernel Equating Using Propensity Scores for Nonequivalent Groups

Wallin¹,

Journal of Educational and Behavioral Statistics

2019

“…The SweSAT is given twice a year (spring and fall) and has traditionally been equated using a matching procedure in which covariates play a major role to assure that similar groups are compared in the equating process. For a description of the equating methods that have been used for the SweSAT, see Lyrén and Hambleton (2011). The five-step kernel equating procedure was carried out in R (R Core Development Team, 2013) using the kernel equating package kequate (Andersson, Bränberg, & Wiberg, 2013).…”

Section: Numerical Illustrationmentioning

confidence: 99%

“…It is important to note that one might argue that an EG design is not suitable here because the groups might be non-equivalent. However, the SweSAT has traditionally used an EG design due to the lack of an anchor test (Lyrén & Hambleton, 2011).…”

Section: Numerical Illustrationmentioning

confidence: 99%

Kernel Equating Under the Non-Equivalent Groups With Covariates Design

Applied Psychological Measurement

Bränberg

2015

When equating two tests, the traditional approach is to use common test takers and/or common items. Here, the idea is to use variables correlated with the test scores (e.g., school grades and other test scores) as a substitute for common items in a non-equivalent groups with covariates (NEC) design. This is performed in the framework of kernel equating and with an extension of the method developed for post-stratification equating in the non-equivalent groups with anchor test design. Real data from a college admissions test were used to illustrate the use of the design. The equated scores from the NEC design were compared with equated scores from the equivalent group (EG) design, that is, equating with no covariates as well as with equated scores when a constructed anchor test was used. The results indicate that the NEC design can produce lower standard errors compared with an EG design. When covariates were used together with an anchor test, the smallest standard errors were obtained over a large range of test scores. The results obtained, that an EG design equating can be improved by adjusting for differences in test score distributions caused by differences in the distribution of covariates, are useful in practice because not all standardized tests have anchor tests.

“…However, the use of external variables seems especially appropriate in two specific contexts. The first is in testing programs not designed or not well suited for equating with an internal anchor test (e.g., Wiberg & Bränberg, 2015), including testing programs that utilize an EG design with groups that may not be randomly equivalent (e.g., Lyren & Hambleton, 2011). The second is in less traditional testing programs, such as those involving small-scale, formative, or longitudinal assessments.…”

Section: Discussionmentioning

confidence: 99%

Linking With External Covariates: Examining Accuracy by Anchor Type, Test Length, Ability Difference, and Sample Size

Albano

Applied Psychological Measurement

2019

Research has recently demonstrated the use of multiple anchor tests and external covariates to supplement or substitute for common anchor items when linking and equating with nonequivalent groups. This study examines the conditions under which external covariates improve linking and equating accuracy, with internal and external anchor tests of varying lengths and groups of differing abilities. Pseudo forms of a state science test were equated within a resampling study where sample size ranged from 1,000 to 10,000 examinees and anchor tests ranged in length from eight to 20 items, with reading and math scores included as covariates. Frequency estimation linking with an anchor test and external covariate was found to produce the most accurate results under the majority of conditions studied. Practical applications of linking with anchor tests and covariates are discussed.