The Problem of Bias in Person Parameter Estimation in Adaptive Testing

Abstract

It is shown that deviations of estimated from true values of item difficulty parameters, caused for example by item calibration errors, the neglect of randomness of item difficulty parameters, testlet effects, or rule-based item generation, can lead to systematic bias in point estimation of person parameters in the context of adaptive testing. This effect occurs even when the errors of the item difficulty parameters are themselves unbiased. Analytical calculations as well as simulation studies are discussed.Keywords computerized adaptive testing, biased point estimation, item response theory, calibration errors, differential item functioning, testlets, computerized item generation As many computer-based methods are readily available nowadays, adaptive testing has become a popular and efficient testing method in psychometrics. The basic principle of an adaptive test can be shortly described as follows: An examinee is first given a few items to obtain a crude initial estimate of the person parameter. Then the next item is chosen so that it contributes maximally to the precision of the updated estimate obtained after the examinee has completed the item. Therefore, several item selection criteria have been developed, for example, the criterion of maximal Fisher information, which is probably the most commonly used in practice (van der Linden, 2010). Furthermore, a number of content-and test-specific constraints often have to be taken into account. The item selection process is iteratively repeated and terminates after a certain predefined number of items or when the estimate of the person parameter no longer changes significantly.Item response theory (IRT) models are an essential element of adaptive testing, as they allow for an easy evaluation of the Fisher information for single items. A central feature of IRT models is that they have separate parameters to describe item and person characteristics. However, in this article it is shown that when the true item difficulty parameters of IRT models differ from the estimated ones, a systematic bias in the estimation of person parameters in adaptive testing arises, even when the errors of the difficulty parameters are unbiased.

Computerized Adaptive Testing in Early Education: Exploring the Impact of Item Position Effects on Ability Estimation

“…Strong positive correlations between true and estimated ability indicate that the relative ordering of test takers changes minimally across conditions, both with and without position effects. This suggests that position bias could be adjusted for with a linear transformation of the theta scale, as noted by Doebler (). Adjustments may be less useful when person or item parameters are not distributed normally, or when position has more than a fixed linear effect.…”

“…Building on the simulation design from Doebler (), this article extends previous research on position effects by addressing the issue of impact in two connected studies. In Study 1, position effects are modeled with operational data from a linear, fixed length assessment used in early education, an area of measurement that has received relatively limited psychometric attention, and one that presents unique and interesting challenges such as more stringent limitations on testing time and test length.…”

“…The goal of Study 2 was to examine the impact of results such as these on a hypothetical subsequent CAT that might employ parameters like the ones from Study 1 as precalibrated values. Similar to Doebler (), CAT was simulated using different sets of generating parameters, where simulated test takers responded based on true item parameters, but the CAT algorithm made decisions based on estimates, as would take place operationally. The design of Study 2 was also informed by practical considerations associated with testing in early education and the use of brief, formative measures of preliteracy skills, such as alphabet knowledge.…”

“…Doebler () outlines how error in item parameter estimation can carry forward to produce bias in person parameter estimation. Within the context of CAT, discrepancies between estimated and true item difficulty are summarized as coming from four main sources: (1) random error due to sampling of test takers, that is, standard error (SE), which tends to be higher in CAT because of higher item turnover and smaller sample sizes; (2) differences across person groups, that is, DIF; (3) testlet effects, a type of context effect that can arise with item sets that share a common stimulus; and (4) automatic item generation or cloning, where variability in item parameters may be ignored for items generated from the same template.…”

“…Within the context of CAT, discrepancies between estimated and true item difficulty are summarized as coming from four main sources: (1) random error due to sampling of test takers, that is, standard error (SE), which tends to be higher in CAT because of higher item turnover and smaller sample sizes; (2) differences across person groups, that is, DIF; (3) testlet effects, a type of context effect that can arise with item sets that share a common stimulus; and (4) automatic item generation or cloning, where variability in item parameters may be ignored for items generated from the same template. In a series of CAT simulation studies, Doebler () manipulated the first and fourth of these sources, with results demonstrating varying amounts of person parameter bias across different IRT models, estimators, test lengths, and item pool sizes. When item difficulty SE was simulated to be .25, mean bias in person ability exceeded .50 logits under some conditions.…”