Item Selection Methods Based on Multiple Objective Approaches for Classifying Respondents Into Multiple Levels

Abstract

Computerized classification tests classify examinees into two or more levels while maximizing accuracy and minimizing test length. The majority of currently available item selection methods maximize information at one point on the ability scale, but in a test with multiple cutting points selection methods could take all these points simultaneously into account. If for each cutting point one objective is specified, the objectives can be combined into one optimization function using multiple objective approaches. Simulation studies were used to compare the efficiency and accuracy of eight selection methods in a test based on the sequential probability ratio test. Small differences were found in accuracy and efficiency between different methods depending on the item pool and settings of the classification method. The size of the indifference region had little influence on accuracy but considerable influence on efficiency. Content and exposure control had little influence on accuracy and efficiency.

Stopping rules for multi‐category computerized classification testing

“…If the maximum test length is reached before the SPRT makes a classification, the person is classified based on the relative location of the MLE and cut‐off points. This method was first applied to CCT by Eggen (1999) and was evaluated later by Thompson (2009) and van Groen, Eggen, and Veldkamp (2014) for the case of two cut‐offs. However, it was argued by Ghosh (1970) that when more than three hypotheses are considered, the Sobel–Wald approach may not be able to lead to a clear decision.…”

“…The value of δ was fixed at 0.2, and the rationale is as follows. Van Groen et al (2014) noted that ‘the size of the indifference region had little influence on accuracy but considerable influence on efficiency’. Similar results were also found in Thompson (2011) as well as Huebner and Fina (2015), in which they compared two levels of , 0.1 and 0.2, and found that leads to considerably shorter test length with nearly no sacrifice of accuracy.…”

Multidimensional Computerized Adaptive Testing for Classifying Examinees

“…This method can be used if just one cutoff point is specified per dimension. Several methods have been developed for item selection with multiple cutoff points for unidimensional classification testing (Eggen and Straetmans 2000;Van Groen et al 2014a;Wouda and Eggen 2009), The weighting method combines the objective functions per cutoff point into one weighted objective function. The weight for the cutoff points depends on the distance of the cutoff point to the current ability estimate (Van Groen et al 2014a).…”

“…MIRT (Reckase 2009) provides such a framework. In a calibrated item bank, model fit is established, item parameter estimates are available, and items with undesired characteristics are removed (Van Groen et al 2014a). During testing, it is assumed that the item parameters have been estimated with enough precision to consider them known (Veldkamp and Van der Linden 2002).…”

A Multidimensional IRT Approach for Dynamically Monitoring Ability Growth in Computerized Practice Environments

“…Maris and Van der Maas (2012) derived a scoring rule that accounts for response time and accuracy, and applied it within ERS. While the ERS can be used to gradually obtain reliable estimates of both student's abilities and item difficulties, adaptive item sequencing can be more efficient if we could start from a pre-calibrated item bank, including information on item difficulty and possibly other characteristics of items, and from which items with undesired characteristics are excluded (van Groen et al, 2014). In this case, trueβ^j(t) in Equation (5) needs not be updated.…”