Evaluating Statistical Targets for Assembling Parallel Mixed‐Format Test Forms

Abstract

Test assembly is the process of selecting items from an item pool to form one or more new test forms. Often new test forms are constructed to be parallel with an existing (or an ideal) test. Within the context of item response theory, the test information function (TIF) or the test characteristic curve (TCC) are commonly used as statistical targets to obtain this parallelism. In a recent study, Ali and van Rijn proposed combining the TIF and TCC as statistical targets, rather than using only a single statistical target. In this article, we propose two new methods using this combined approach, and compare these methods with single statistical targets for the assembly of mixed‐format tests. In addition, we introduce new criteria to evaluate the parallelism of multiple forms. The results show that single statistical targets can be problematic, while the combined targets perform better, especially in situations with increasing numbers of polytomous items. Implications of using the combined target are discussed.

Multidimensional Test Assembly Using Mixed-Integer Linear Programming: An Application of Kullback–Leibler Information

“…First, as could be expected giving previous research (cf. Ali & van Rijn, 2016; Debeer et al, 2017), using the TCC as the statistical target results in a better performance with respect to the TCC, but a worse performance with respect to the error variance. Second, for all the other Minimax models the difference in performance with respect to the error variance is limited, but the combined approach with additional TCC constraints performs better with respect to TCC parallelism (cf.…”

“…Second, for all the other Minimax models the difference in performance with respect to the error variance is limited, but the combined approach with additional TCC constraints performs better with respect to TCC parallelism (cf. Ali & van Rijn, 2016; Debeer et al, 2017). Third, increasing the number of θ -points from three to five improves the parallelism both with respect to the error variance and the TCC.…”

“…Using simulated item pools, of which some were based on operational items, the performance of the proposed statistical targets was compared in a uni-, a two-, and a three-dimensional bifactor case. The results indicated that (a) the KLIs can replace the traditional TIF as statistical target in the unidimensional case, (b) the proposed statistical targets can be successfully applied in two- and three-dimensional cases (note that the linear approximations are limited to the two-dimensional case), (c) the combined approach with additional TCC constraints (Ali & van Rijn, 2016; Debeer et al, 2017) outperforms the single approach regardless of the item pool or the chosen statistical target, and (d) in contrast with the unidimensional case, using only the TCC as the statistical target also results in good parallelism with respect to the error variance in the multidimensional case.…”

“…so that the TCCs of the assembled test forms were constrained to be within an absolute distance of y h T from the target TCC at θ -point h (i.e., one of the combined approaches proposed by Debeer et al, 2017). In Equation 13 y h T was set to 0 . 25 at all θ -points.…”

“…Similarly, with the TCC as statistical target (i.e., parallel expected test scores), P j ( boldθ h ) and P ( t ) ( boldθ h ) are used. Note that when using the TCC, it is more logical to minimize the maximal absolute, rather than the relative difference (Debeer, Ali, & van Rijn, 2017).…”