Determining the Length of a Criterion-Referenced Test

Abstract

When determining how many items to include on a criterion-referenced test, practitioners must resolve various nonstatistical issues before a particular solution can be applied. A fundamental problem is deciding which of three true scores should be used. The first is based on the probability that an examinee is correct on a "typical" test item. The second is the probability of having acquired a typical skill among a domain of skills, and the third is based on latent trait models. Once a particular true score is settled upon, there are several perspectives that might be used to determine test length. The paper reviews and critiques these solutions. Some new results are described that apply when latent structure models are used to estimate an examinee's true score. When trying to determine how many items to include on a criterion-referenced test, perhaps the most fundamental problem is that there are at least three conceptualizations, or models, of an achievement test that might be used. Each of these conceptualizations is based on a different type of true score. The first deals with the number of items an examinee would get correct if he/she were to respond to every item in some item domain. The second is concerned with the proportion of skills among a domain of skills that an examinee has acquired. Because of errors at the item level, such as guessing, this conceptualization is different from the first. The final approach is based upon latent trait models. In some cases, one model might yield substantially different results from another in terms of test length, and so the choice of a model can be crucial. Once one of the above conceptualizations is settled upon, a variety of other issues must be resolved. For example, when comparing an examinee's true score to a standard, should it be assumed that the standard is known, or should the process by which it was determined be taken into account? Should the test length problem be formulated in terms of a single examinee, a &dquo;typical&dquo; examinee, or both? How certain do we want to be of making a correct decision (classification) of an examinee? Are we willing to use a Bayesian solution? This paper has several goals. The first is to give a brief review and critique of the three general approaches that might be used when determining the length of a criterion-referenced test. The second is to describe new results on test length, and the third is to indicate possible directions for future research.

The Single Administration Estimate of the Proportion of Agreement of a Proficiency Test Scored with a Latent Structure Model

“…One appealing feature of this method of scoring a test, particularly for the case of multiple-choice test items, is that the resulting estimate of true score is corrected for guessing without assuming guessing is at random. In addition, results reported by Wilcox (1980) suggest that for some situations, scoring proficiency tests with latent structure models might sub- stantially reduce the number of items on the test that would otherwise be needed to obtain the same level of accuracy when comparing ~ tõ o If, however, latent structure models are used to score a test, existing, single administration estimates of reliability are no longer appropriate because, as will become evident, we must estimate a multinomial distribution rather than the simpler binomial probability function. The purpose of this paper is to describe and compare various solutions to this problem.…”

“…If f is greater than or equal to some predetermined constant, ro, the examinee is said to have passed the test; otherwise, he/she fails. For comments on how to determine ro, see Hambleton, et al 1978), Glass (1978), Wilcox (1980). One approach to characterizing such tests is to estimate the probability that for the typical examinee, ~ > ro when ro or that r < ~o when ~ < roo For the special case of one item per skill (i.e., k = 1), these probabilities can be estimated when the binomial error model (e.g., Lord and Novick, 1968, Chapter 23) or the Dirichlet-multi-…”

“…However, for the case of k >_ 2 items per skill, a strong true-score model has not been derived that can be used to obtain point estimates of these probabilities. However, as noted by Wilcox (1980), bounds on these probabilities can be derived and estimated.…”

Sampling knowledge and understanding: how long should a test be?

“…Thus the course may cover many distinct topics and involve rote learning and complex argument, calculation and non-quantitative description, lectures and laboratory work, good teaching and under-emphasis. The Rasch model, and item response theory generally, are then inappropriate (Wilcox, 1980;Burton, 2004b). Indeed, typical academic tests of the kinds discussed here conform much more to the Posey model even though the facts and ideas to be tested are usually neither independent nor equivalent in difficulty and utility.…”

“…Indeed, individual items may differ in inherent difficulty, and student knowledge may be patchy. However, there is one particular situation in which these complications are irrelevant, namely that in which the knowledge domain is sampled in a completely random way (Wilcox, 1980). Few tests do this.…”