Effect of Examinee Ability on Test Equating Invariance

Abstract

Previous research on the application of IRT methodology to vertical test equating has demonstrated conflicting results about the degree of invariance shown by these methods with respect to examinee ability. The purpose of this study was to examine IRT equating invariance by simulating the vertical equating of two tests under varying conditions. Rasch, three-parameter, and equipercentile equating methods were compared. Six equating cases, using different sets of item parameters, were replicated based on examinee samples of low, medium, or high ability or where ability was matched to the difficulty level of the test. The results showed that all three methods were reasonably invariant to examinee ability level under all conditions imposed. This suggests that multidimensionality is likely to be the cause of the lack of invariance found in real datasets.The equating of scores on two or more tests is fundamental to large testing programs. Scores on new forms of tests must be equated to scores on previous versions of the same test; this type of equating is typically referred to as &dquo;horizontal because new test forms are designed to be as similar as possible to previous forms in terms of their psychometric properties. The need for a different kind of equating, called 'b~~rt~c~l&dquo; equating, arises when it is desired to link scores across levels of achievement tests. These levels are likely to be targeted to grades in school, and tests corresponding to different levels differ intentionally both in content and difficulty. Obviously, from theoretical and operational perspectives, vertical equating is much more difficult to accomplish than horizontal equating.In recent years, a considerable quantity of research has investigated the application of item response theory (IRT) methodology to both types of test equating. Although a large number Of IRT models have been proposed, the research on test equating has been primarily limited to two models-the one-parameter logistic (Rasch) model and the three-parameter model.According to the Rasch model, the probability of an examinee with ability 0 (defined generically) answering item correctly is solely a function of the item's difficulty (bi) and the e~c~mi~~~'s ability (0):The three-parameter model incorporates two additional item parameters:at UNIV CALIFORNIA SANTA BARBARA on July 4, 2015 apm.sagepub.com Downloaded from

Comparative Analysis of Classical Test Theory and Item Response Theory Based Item Parameter Estimates of Senior School Certificate Mathematics Examination

“…They noted that "because IRT differs considerably from CTT in theory, and commands some crucial theoretical advantages over CTT, it is reasonable to expect that there would be appreciable differences between the IRT and CTT-based item and person statistics". However, findings by (Lawson, 1991;Skaggs & Lissitz, 1988;Stage, 1999) have all found differences between IRT and CTT theory estimates. On the other hand, some other researchers noted that the agreement between results from item-analyses performed within the two different frameworks, IRT and CTT, was very good.…”

Exploring the Full-Information Bifactor Model in Vertical Scaling With Construct Shift

“…For example, Skaggs and Lissitz (1988) suggested that reading and vocabulary tests might be more unidimensional across grades or may provide more invariant vertical scaling results. Wang and Jiao (2009) conducted an empirical study using multi-group confirmatory factor analysis (CFA) and found evidence for construct invariance across grades in a vertical scale for a K-12 large-scale reading test.…”

Monte Carlo Studies in Item Response Theory

“…For example, parameter recovery studies for the 2PLM would naturally use this as the underlying mathematical model in the study (e.g., Drasgow, 1989;Harwell & Janosky, 1991; Reise & Yu, 1990). Similar comments hold for other IRT-based research settings that use t~'r models whose implementation depends on the nature of the problem; for example, studies of equating methods (e.g., Baker & Al-l~arr~i, 1991;Skaggs & Lissitz, 1988), dimensionality (e.g., Ackerman, 1989;Nandakumar, 1991), perse~r~litem goodness of fit (e.g., Holt & Macready, 1989;Mcl~inl~y ~ Mills, 1985), adaptive testing/computerized testing (e.g., De Ayala, Dodd, & Koch, 1990)9 test speededness (e.g., Oshima, 1994), and criterion-referenced assessment (e.g., Plake & Kane, 1991 ). A particularly large class of modeling problems involves DIF in which the model underlying data generation must be justified with regard to the number of DIF items, the parameters that will reflect DIF and the degree of DIP, whether group differences are modeled, and how the matching criteria should be calculated (e.g., Candell & Drasgow, 1988;Gifford & Swaminathan, 1990 (Baker, 1989) to generate dichotomous item response data following a 2PLM, BILOG to estimate IRT model parameters, and a lodeveloped program to compute ~l~ts~s.…”