No abstract

The Norz-Equivalent-groups Anchor Test (NEAT) design has been in wide use since at least the early 1940s. It involves two populations of test takers, P and Q, and makes use of an anchor test to link them. Two linking methods used for NEAT designs ure those ( a ) based on chain quating and (b) that use the anchor test to post-strati& the distributions of the two operational test scores to a common population (i.e., Tucker equating and frequency estimation). We show that, under digwent sets of assumptions, both methods are observed score equuting methods and we give conditions under which the methods give identical results. In addition, we develop analogues of the Doratis and Holland (2000) RMSD measures of population invariance of equating methods for the NEAT design,for both chain and Vost-stratification equating methods.

This study examines how closely the kernel equating (KE) method (von Davier, Holland, & Thayer, 2004a) approximates the results of other observed-score equating methodsequipercentile and linear equatings. The study used pseudotests constructed of item responses from a real test to simulate three equating designs: an equivalent groups (EG) design and two non-equivalent groups with anchor test (NEAT) designs, one with an internal anchor and another with an external anchor. To compare results, the study sets the equating function in the EG design as the equating criterion. In these examples, the KE results were very close to the results from the other equating methods. Moreover, in almost all situations investigated, the KE results were closer to the equating criterion.

This study addressed the sampling error and linking bias that occur with small samples in a nonequivalent groups anchor test design. We proposed a linking method called the synthetic function, which is a weighted average of the identity function and a traditional equating function (in this case, the chained linear equating function). Specifically, we compared the synthetic, identity, and chained linear functions for various‐sized samples from two types of national assessments. One design used a highly reliable test and an external anchor, and the other used a relatively low‐reliability test and an internal anchor. The results from each of these methods were compared to the criterion equating function derived from the total samples with respect to linking bias and error. The study indicated that the synthetic functions might be a better choice than the chained linear equating method when samples are not large and, as a result, unrepresentative.

New technology enables interactive and adaptive scenario-based tasks (SBTs) to be adopted in educational measurement. At the same time, it is a challenging problem to build appropriate psychometric models to analyze data collected from these tasks, due to the complexity of the data. This study focuses on process data collected from SBTs. We explore the potential of using concepts and methods from social network analysis to represent and analyze process data. Empirical data were collected from the assessment of Technology and Engineering Literacy, conducted as part of the National Assessment of Educational Progress. For the activity sequences in the process data, we created a transition network using weighted directed networks, with nodes representing actions and directed links connecting two actions only if the first action is followed by the second action in the sequence. This study shows how visualization of the transition networks represents process data and provides insights for item design. This study also explores how network measures are related to existing scoring rubrics and how detailed network measures can be used to make intergroup comparisons.