Does Preequating Work? An Investigation Into a Preequated Testlet‐based College Placement Exam Using Postadministration Data

Abstract

In this study, we investigated whether preequating results agree with equating results that are based on observed operational data (postequating) for a college placement program. Specifically, we examined the degree to which item response theory (IRT) true score preequating results agreed with those from IRT true score postequating and from observed score equating. Three academic subjects were examined in this study: analyzing and interpreting literature, American government, and college algebra. The findings suggested that differences between equating results from IRT true score preequating and postequating varied from subject to subject. In general, IRT true score postequating agreed with IRT true score preequating for most of the forms for a test subject. Any difference among the equating results can be attributed to the way through which items were pretested, contextual/order effects, or the violation of IRT assumptions.Key words: IRT preequating, IRT true score equating, observed score equating, IRT postequating 1 Equating is a statistical process used to adjust scores on two or more forms of a test so that the scores can be used interchangeably (Kolen & Brennan, 2004). Depending on when equating is conducted, it can be further categorized as preequating and postequating.Preequating, as the name implies, refers to the process through which conversions from raw to scaled scores are established prior to the time the new test is administered operationally as an intact final form. Preequating often is based on item response theory (IRT).Due to the fact that preequating can establish the conversion table prior to operational testing, there are a number of advantages in using preequating rather than postequating (see Eignor, 1985;Kirkpatrick & Way, 2008;and Kolen & Brennan, 2004, for a complete review).These advantages include more flexible assessment and a better quality control check for the tests. Perhaps the most appealing feature of preequating is the ability to facilitate immediate score reporting for tests that require reporting scores immediately after the test administration.tests are good examples. Due to these advantages, preequating is sometimes used in large-scale assessments.Much research has been conducted to compare the difference in equating results between preequating and postequating. In general, results have not been consistent. Eignor (1985), in investigating the feasibility and practical outcomes of preequating SAT ® verbal and mathematical sections through IRT true score equating, found that preequating worked adequately for the verbal sections but not for the mathematical section. Similarly, Kolen and Harris (1990), in comparing item preequating and random groups equating using IRT and equipercentile methods, found that preequating performed poorly for the ACT math test. Contributing reasons for poor preequating results mainly focused on the inconsistent behaviors of test items in pretest and operational contexts. These inconsistencies can be caused by a lack of motivation on the p...

Preequating With Empirical Item Characteristic Curves: An Observed‐Score Preequating Method

“…Despite these mixed results, IRT preequating continues to be one of the most sought‐after methods for preequating. For example, large‐scale testing programs such as the College‐Level Examination Program® (CLEP®) or Graduate Record Examinations® (GRE®) use IRT to preequate new test forms (Gao, He, & Ruan, ).…”

Evaluating the Accuracy of the Empirical Item Characteristic Curve Preequating Method in the Presence of Test Speededness

“…Potential factors contributing to its uneven performance include context effects, test speededness, item position effects, and multidimensionality. Even so, IRT-based item preequating is still in demand in large-scale testing programs (Zu & Puhan, 2014;Gao et al, 2012). EICC Preequating.…”

“…For instance, Bejar and Wingersky (1982) assessed the feasibility of IRT preequating based on the Test of Standard Written English (TSWE); Eignor and colleagues conducted a series of studies on IRT preequating based on data collected from the SAT (Eignor, 1985; Eignor & Cook, 1983; Eignor & Stocking, 1986; Stocking & Eignor, 1986); and Kolen and Harris (1990) experimented with item preequating in the American College Test’s (ACT) Mathematics Assessment Test. The most recent research on the feasibility of IRT preequating in the context of large-scale testing programs is seen in Domaleski (2006); Tong, Wu, and Xu (2008); and Gao, He, and Ruan (2012). The studies show that IRT-based preequating achieves mixed results.…”

Contributions to Score Linking Theory and Practice

“…While the SAT rejected the use of preequating on the basis of this research, during the 1990s, other testing programs moved to test administration and scoring designs, such as computer-adaptive testing, that relied on even more restrictive invariance assumptions than those that did not hold in the SAT studies. Gao et al (2012) investigated whether IRT true-score preequating results based on a Rasch model agreed with equating results based on observed operational data (postequating) for CLEP. The findings varied from subject to subject.…”