Sampling Variability and Axioms of Classical Test Theory

Abstract

Many well-known equations in classical test theory are mathematical identities in populations of individuals but not in random samples from those populations. First, test scores are subject to the same sampling error that is familiar in statistical estimation and hypothesis testing. Second, the assumptions made in derivation of formulas in test theory are not necessarily satisfied in small samples. The present study derived modified equations relating test scores and components of scores that are identities in samples of any size and that reduce to the more familiar equations when various correlations are zero. Simulations determined the accuracy of both the familiar and the modified equations when applied to samples of various sizes from populations with known reliability coefficients. The programs also determined the variability of the sample values for different parameters in the equations and for different sample sizes, as well as the means and variances of discrepancies between population and sample values.

Neo-Classical Test Theory: A Modernization

“…Novick designated classical test theory as weak true-score theory in that no specific assumptions were made as to the form of observed, true score, or error score distributions. Subsequent attempts have been made to strengthen the theory by deriving its axioms from concepts of probability theory [9,10].…”