The Chain and Post‐Stratification Methods for Observed‐Score Equating: Their Relationship to Population Invariance

ETS and the ETS logo are registered trademarks of Educational Testing Service (ETS). AbstractKernel equating (KE) is a new approach to observed-score equating and is described in detail in von Davier, Holland, and Thayer (2004b). Over the past months, several evaluation studies of KE have been designed and carried out. In this part of the overall evaluation study, we compared the KE method with other equating methods using real data from the program of Praxis Series:Professional Assessments for Beginning Teachers®. The goal of this study was to check how closely the KE results agree with the results from other observed-score equating methods in different operational settings. The equating designs involved include an equivalent-groups (EG) design and two nonequivalent groups with anchor test (NEAT) designs with different sample sizes and different characteristics. We compared KE with the equating methods that were used operationally. The results showed that the differences between KE and the traditional equating methods are very small in the EG design, especially in the linear case. In the two NEAT designs, the KE version of poststratification equating with optimal bandwidths produced close results to its analogue, frequency estimation equipercentile equating, except at the lower score range. The KE linear method yielded very similar results to the Tucker method.

Section: Neat Designmentioning

confidence: 99%

Section: Appendix a An Outline Of Kementioning

confidence: 99%

See 1 more Smart Citation

Comparisons of the Kernel Equating Method With the Traditional Equating Methods on Praxis™ Data

Mao

Davier

Rupp

2006

“…Harris and Kolen (1990) suggested that the PSE and the IRT OSE methods have superior theoretical underpinnings compared to the CE method while the latter method requires the least amount of computation. von Davier, Holland, and Thayer (2004a) provided some theoretical analyses of the PSE and CE methods and showed that they may both be interpreted as examples of OSE methods and that they can provide identical results under special conditions.…”

Section: Tablementioning

confidence: 99%

The Missing Data Assumptions of the Nonequivalent Groups With Anchor Test (Neat) Design and Their Implications for Test Equating

Sinharay

Holland

2009

Self Cite

The nonequivalent groups with anchor test (NEAT) design involves missing data that are missing by design. Three popular equating methods that can be used with a NEAT design are the poststratification equating method, the chain equipercentile equating method, and the itemresponse-theory observed-score-equating method. These three methods each make different assumptions about the missing data in the NEAT design. Though studies have compared the equating performance of the three methods under the NEAT design, none has examined the missing data assumptions and their implications for such comparisons. The missing data assumptions can affect equating studies because it is necessary to fill in the missing data or their distribution in some way in order to have a true, or criterion, equating function to compare the accuracy and bias of the different methods. If the missing data or their distribution are filled in using missing data assumptions that correspond to a given method, that may favor that method in any comparison with the others. This paper first describes the missing data assumptions of the three equating methods and then performs a fair comparison of the 3 methods using data from 3 different operational tests. For each data set, we examine how the 3 equating methods perform when the missing data satisfy the assumptions made by only 1 of these equating methods. The chain equating method is somewhat more satisfactory overall than the other methods in our fair comparison of the methods; hence, we recommend that equating practitioners seriously consider the chain equating method when using the NEAT design. In addition, we conclude that the results from the different equating methods will tend to agree with each other when proper equating conditions are in place. Moreover, to uncover problems that might not reveal themselves otherwise, it is important for operational testing programs to apply multiple equating methods and study the differences among their results.

“…von Davier, Holland, and Thayer (2004) defined the RMSD for the anchor test or NEAT data-collection design, and Holland (2003) defined the REMSD. The REMSD index was used to obtain a single value summarizing the values of RMSD(x) over the distribution of x in the total group.…”

Section: Deviance Measuresmentioning

confidence: 99%

Effect of Repeaters on Score Equating in a Large‐scale Licensure Test

Kim

Walker

2009

This study investigated the subgroup invariance of equating functions for a licensure test in the context of a nonequivalent groups with anchor test (NEAT) design. Examinees who had taken a new, to-be-equated form of the test were divided into three subgroups according to their previous testing experience: (a) repeaters who previously took the reference form, to which the new form would be equated; (b) repeaters who previously took any form other than the reference form; and (c) first-time test takers for whom the new form was the first exposure to the test. Equating functions obtained with two subgroups, all repeaters, and first-time test takers were similar to those obtained with the total group, supporting score equatability of the two forms. However, when the repeater subgroup was divided into two groups based on the particular form examinees took previously, subgroup equating functions substantially differed from the total-group equating function, indicating subgroup dependency of score equating. The results indicate that repeater membership needs to be more clearly specified to assess the impact of repeaters on score equating. Such clarification may be especially necessary for high-stakes licensure tests because repeaters tend to perform more poorly on such tests than first-time test takers.