A General Approach to Algorithmic Design of Fixed-Form Tests, Adaptive Tests, and Testlets

Abstract

The selection of items from a calibrated item bank for fixed-form tests is an optimal test design problem; this problem has been handled in the literature by mathematical (1971, 1980) and Weiss (1976, 1978) The purpose of administering achievement tests is to obtain an efficient estimate of the trait levels of a sample of examinees. The trait estimates obtained from different test types, however, may differ in efficiency. The purpose of the present study was to stress the similarities in designing fixed-form tests, adaptive tests, and testlets using optimal design theory. Reviews of optimal design theory are given by Silvey (1980) and Ford, Kitsos, & Titterington (1989).

An NCME Instructional Module on Multistage Testing

“…In sum, using item sets allows for test developers to have more control over the test process and quality and provides many advantages and relatively few disadvantages when compared with item‐level adaptive and linear tests. Multistage tests seemingly combine the advantages of adaptive testing with the advantages of linear tests (Berger, 1994), providing a balanced compromise between these two test forms. This has led to the increased research and use of multistage tests.…”

“…This results in fewer adaptation points than with item-level computerized adaptive tests (CATs), in which adaptation occurs after every item, but more adaptation points than in conventional paper-and-pencil linear tests, in which all examinees are administered all of the same questions. Multistage tests combine the advantages of both adaptive and linear test forms (Berger, 1994). As such, multistage tests, both two-stage and testlet-based, provide a balanced compromise between computerized adaptive tests and linear tests, which has led to their increasingly widespread research and use.…”

Optimal Design in Educational Testing

“…Low discrimination items have fairly flat information, so that for a normal distribution of setting the difficulties close to the centre gives the most information. Berger (1994b) showed how test design for fixed-form tests, adaptive tests and testlets can all be placed into the same optimal design framework. Testlets are collections of items developed as a small test with the idea of simplifying some aspects of adaptive testing.…”

Minimax D-Optimal Designs for Item Response Theory Models

“…These approaches actually involve optimization over an empirical item pool. Berger (1994b) proposed a general framework for the construction of fixed form tests, adaptive tests and testlets by means of an unconstrained nonlinear approach and Berger and Mathij ssen (1997) expanded these results to polytomous IRT models. The former approaches involve optimization over a parameter space.…”

“…One approach is to construct the design sequentially, that is, to use information from previous runs (Wynn, 1970;Wu, 1985). This approach was applied to dichotomous IRT models by Berger (1992Berger ( , 1994aBerger ( , 1994b and Jones and Jin (1994). Although these designs have been applied successfully in educational and psychological measurement, in some cases these designs may sometimes be too expensive and not practical in classrooms where, for example, a whole class of children is tested simultaneously.…”