Classical Test Theory as a First‐order Item Response Theory: Application to True‐score Prediction From a Possibly Nonparallel Test

Abstract

We give an account of classical test theory (CTT) in terms of the more fundamental ideas of item response theory (IRT). This approach views CTT as a very general version of IRT, and the commonly used IRT models as detailed elaborations of CTT for special purposes. We then use this approach to CTT to derive some general results regarding the prediction of the true score of a test from an observed score on that test as well from an observed score on a different test. This leads us to a new view of linking tests that were not developed to be linked to each other. In addition we propose true-score prediction analogues of the Dorans and Holland measures of the population sensitivity of test linking functions. We illustrate the accuracy of the first-order theory using simulated data from the Rasch model and illustrate the effect of population differences using a set of real data.Key words: test theory, true scores, best linear predictors, test linking, nonparallel tests, simulation, Rasch Model ii AcknowledgementsWe would like to thank Neil Dorans, Skip Livingston, and two anonymous referees for many suggestions that have greatly improved this paper. The work reported here is collaborative in every respect and the order of authorship is alphabetical. It was begun while both authors were on the faculty at the

Relationships of Measurement Error and Prediction Error in Observed‐Score Regression

“…The results of (6)–(12) results showing how an observed‐score regression's total prediction error variance can be decomposed into the sum of parts due to true score variance and error variance, expand on the results of prior discussions of prediction error (Cochran, 1970; Holland & Hoskens, 2003; Lord & Novick, 1968). To relate this paper's results to prior discussions that have focused on one predictor, the single predictor versions of (6) and (10)–(12) are shown in Table 1 (where the observed and true score regression slopes can be calculated as and ).…”

“…The implications of measurement error in observed‐score regressions of test scores are usually described by incorporating classical test theory test theory assumptions into results for a regression involving a criterion test, Y , and one predictor test, X 1 (Haberman, 2008; Holland & Hoskens, 2003; Lord & Novick, 1968). These classical test theory assumptions are presented here in more general terms, where the observed scores of Y and J ≥ 1 predictor tests, X 1 , …, X J , are expressed as sums of their true scores and errors, …”

“…The focus of prior discussions usually has been on how measurement errors E Y and E X 1 , … … , E XJ , affect the accuracy of the regression based on observed scores for approximating the results of regressions involving true scores, such as , and (Cochran, 1968, 1970; Glass, Peckham, & Sanders, 1972; Haberman, 2008; Holland & Hoskens, 2003; Lindley, 1947; Linn & Werts, 1971; Lord, 1957, 1960, 1962; Madansky, 1959; Schumacker & Lomax, 1996). These prior discussions are particularly attentive to the measurement error(s) of the Xs which are a direct violation of the usual observed‐score regression assumption that the Xs have no measurement error and which result in observed‐score regressions that differ from true score regressions.…”

“…Although prior discussions of how measurement errors affect the accuracy of observed‐score regressions are extensive (Cochran, 1968, 1970; Glass, Peckham, & Sanders, 1972; Haberman, 2008; Holland & Hoskens, 2003; Lindley, 1947; Linn & Werts, 1971; Lord, 1957, 1960, 1962; Madansky, 1959; Pedhazur, 1997; Schumacker & Lomax, 1996), these discussions have not directly addressed every question that arises about the prediction error of an observed‐score regression function. The prior discussions have tended to give primary focus to how measurement errors reduce the accuracy of the observed‐score regression function for approximating the true score regression function while treating prediction error implications as a secondary focus.…”

Item Response Theory

“…Xu and von Davier (2008c) developed an IRT linking approach for use with the GD model and applied the proposed approach to NAEP data. Holland and Hoskens (2002) developed an approach viewing CTT as a firstorder version of IRT and the latter as detailed elaborations of CTT, deriving general results for the prediction of true scores from observed scores, leading to a new view of linking tests not designed to be linked. They illustrated the theory using simulated and actual test data.…”