Evaluating Comparability in Computerized Adaptive Testing: Issues, Criteria and an Example

Abstract

When a computerized adaptive testing (CAT) version of a test co‐exists with its paper‐and‐pencil (P&P) version, it is important for scores from the CAT version to be comparable to scores from its P&P version. The CAT version may require multiple item pools for test security reasons, and CAT scores based on alternate pools also need to be comparable to each other. In this paper, we review research literature on CAT comparability issues and synthesize issues specific to these two settings. A framework of criteria for evaluating comparability was developed that contains the following three categories of criteria: validity criterion, psychometric property/reliability criterion, and statistical assumption/test administration condition criterion. Methods for evaluating comparability under these criteria as well as various algorithms for improving comparability are described and discussed. Focusing on the psychometric property/reliability criterion, an example using an item pool of ACT Assessment Mathematics items is provided to demonstrate a process for developing comparable CAT versions and for evaluating comparability. This example illustrates how simulations can be used to improve comparability at the early stages of the development of a CAT. The effects of different specifications of practical constraints, such as content balancing and item exposure rate control, and the effects of using alternate item pools are examined. One interesting finding from this study is that a large part of incomparability may be due to the change from number‐correct score‐based scoring to IRT ability estimation‐based scoring. In addition, changes in components of a CAT, such as exposure rate control, content balancing, test length, and item pool size were found to result in different levels of comparability in test scores.

Validity in Computer-Based Testing: a Literature Review of Comparability Issues and Examinee Perspectives

“…Wang & Kolen (2001) illustrated that if a score of 18 on the ACT Assessment scale score is used as a cut score for collegiate sports eligibility by the NCAA, there were 71% of the examinees who had P&P scores of 18 or above while there were 68% of the examinees who got CAT scores of 18 or above. If the CAT scores were used due to the CAT reliability over the P&P test, then it would be only 3% fewer examinees in the CAT compared to the examinees who took the P&P tests that were eligible for collegiate sports.…”

“…The findings indicated that cumulative scores distribution in computerized adaptive tests are quite similar to one another, but they differed from the paperand-pencil tests scale scores. Wang & Kolen (2001) assumed that the differences in scoring methods might influence the major difference in scale score distribution for both test versions. Gallagher, Bennett, Cahalan, & Rock (2002) examined a computerized Mathematical Expression (ME) using ANOVA to detect construct-irrelevant variance.…”

“…While the computer delivery algorithm, either linear or adaptive computer tests, did not result in any differences between CBT and P&P scores. Wang & Kolen (2001) addressed comparability analyses between ACT Mathematics Assessment computerized adaptive version and paper-and-pencil version through simulation procedures. Raw scores from both versions were converted into the ACT Assessment scale score (range from 1 to 36) to better understand the score interpretation.…”

“…One clear ad-vantage of the CBT administration, particularly in adaptive version (CAT), is to reduce testing time while maintaining the test reliability over P&P versions (Wang & Kolen, 2001). …”

Számítógép alapú adaptív és lineáris tesztek összehasonlító hatékonyságvizsgálata

“…Diákonkénti bontásban a lineáris, illetve az adaptív tesztkörnyezetben történt képességszint-becslés során elkövetett hiba nagyságát összehasonlítva (Wang és Kolen, 2001;Wang, 2010) megállapítható, hogy a lineáris formátumú teszt alapján történt képességszint-becslés hibáinak nagysága diákszinten átlagosan nagyobb (t=-7,54, p<0,01; se_átlag=0,53), mint ugyanazon diákok adaptív tesztkörnyezetben történt képességszint-becslésének hibája (se_átlag=0,49). A hatás-mérték 0,78, vagyis jelentős az eltérés.…”