No abstract

In this study, we investigated whether preequating results agree with equating results that are based on observed operational data (postequating) for a college placement program. Specifically, we examined the degree to which item response theory (IRT) true score preequating results agreed with those from IRT true score postequating and from observed score equating. Three academic subjects were examined in this study: analyzing and interpreting literature, American government, and college algebra. The findings suggested that differences between equating results from IRT true score preequating and postequating varied from subject to subject. In general, IRT true score postequating agreed with IRT true score preequating for most of the forms for a test subject. Any difference among the equating results can be attributed to the way through which items were pretested, contextual/order effects, or the violation of IRT assumptions.Key words: IRT preequating, IRT true score equating, observed score equating, IRT postequating 1 Equating is a statistical process used to adjust scores on two or more forms of a test so that the scores can be used interchangeably (Kolen & Brennan, 2004). Depending on when equating is conducted, it can be further categorized as preequating and postequating.Preequating, as the name implies, refers to the process through which conversions from raw to scaled scores are established prior to the time the new test is administered operationally as an intact final form. Preequating often is based on item response theory (IRT).Due to the fact that preequating can establish the conversion table prior to operational testing, there are a number of advantages in using preequating rather than postequating (see Eignor, 1985;Kirkpatrick & Way, 2008;and Kolen & Brennan, 2004, for a complete review).These advantages include more flexible assessment and a better quality control check for the tests. Perhaps the most appealing feature of preequating is the ability to facilitate immediate score reporting for tests that require reporting scores immediately after the test administration.tests are good examples. Due to these advantages, preequating is sometimes used in large-scale assessments.Much research has been conducted to compare the difference in equating results between preequating and postequating. In general, results have not been consistent. Eignor (1985), in investigating the feasibility and practical outcomes of preequating SAT ® verbal and mathematical sections through IRT true score equating, found that preequating worked adequately for the verbal sections but not for the mathematical section. Similarly, Kolen and Harris (1990), in comparing item preequating and random groups equating using IRT and equipercentile methods, found that preequating performed poorly for the ACT math test. Contributing reasons for poor preequating results mainly focused on the inconsistent behaviors of test items in pretest and operational contexts. These inconsistencies can be caused by a lack of motivation on the p...

No abstract

Successful detection of unusual responses is critical for using machine scoring in the assessment context. This study evaluated the utility of approaches to detecting unusual responses in automated essay scoring. Two research questions were pursued. One question concerned the performance of various prescreening advisory flags, and the other related to the degree of machine scoring difficulty and whether the size of the human–machine discrepancy could be predicted. The results suggested that some advisory flags operated more consistently across measures and tasks in detecting responses that the machine was likely to score differently from human raters than did other flags, and relatively little scoring difficulty was found for three of the four tasks examined in this study, with the relationship between machine and human scores being reasonably strong. Limitations and future studies are also discussed.