Developing Multistage Tests Using D-Scoring Method

Abstract

The D-scoring method for scoring and equating tests with binary items proposed by Dimitrov offers some of the advantages of item response theory, such as item-level difficulty information and score computation that reflects the item difficulties, while retaining the merits of classical test theory such as the simplicity of number correct score computation and relaxed requirements for model sample sizes. Because of its unique combination of those merits, the D-scoring method has seen quick adoption in the educational and psychological measurement field. Because item-level difficulty information is available with the D-scoring method and item difficulties are reflected in test scores, it conceptually makes sense to use the D-scoring method with adaptive test designs such as multistage testing (MST). In this study, we developed and compared several versions of the MST mechanism using the D-scoring approach and also proposed and implemented a new framework for conducting MST simulation under the D-scoring method. Our findings suggest that the score recovery performance under MST with D-scoring was promising, as it retained score comparability across different MST paths. We found that MST using the D-scoring method can achieve improvements in measurement precision and efficiency over linear-based tests that use D-scoring method.

Latent D-Scoring Modeling: Estimation of Item and Person Parameters

“…The DSM-C is a classical approach to test scoring, equating, and item analysis, with (a) the person’s ability score, D w , being a weighted sum of the observed item scores (see Equation 1) and (b) an analytic IRF model (see Equation 2) used as a nonlinear regression for the prediction of item parameters from the person scores, D w , computed a priori via Equation (1). The reader may refer to previous studies on the DSM-C method and its application for polytomous items, multistage testing, and other measurement scenarios (e.g., Dimitrov, 2017, 2019; Dimitrov & Luo, 2018; Han et al, 2019).…”

Modeling of Item Response Functions Under the D-Scoring Method

“…In some practical applications of the LRM on the D -scale, the underestimation of item true scores has been avoided by using inverse-regression adjustments (e.g., Han et al, 2018), but this procedure is too complicated for regular applications of the DSM. To solve this problem, the present study offers new IRF models, referred to as rational function models (RFMs), with one parameter (RFM1), two parameters (RFM2), and three parameters (RFM3).…”

“…A recently developed method of scoring and analyzing test data, referred to as D-scoring method (DSM; Dimitrov, 2016, 2017), is adopted in the assessment practice at the National Center for Assessment in Saudi Arabia and gaining attention in the field of educational and psychological measurement. The transparency, simplicity, and efficiency of the DSM have been demonstrated in numerous studies (e.g., Al-Mashary & Dimitrov, 2018; Dimitrov, 2016, 2017; Domingue & Dimitrov, 2015; Dimitrov & Luo, 2017a, 2017b, 2018; Han, Dimitrov, & Al-Mashary, 2018) and computerized applications such as the System for Automated Test Scoring and Equating Under the D-scoring Method (SATSE-D, Atanasov & Dimitrov, 2019a), System for Automated Test Assembly (SATA-D; Atanasov & Dimitrov, 2019b), and a self-sustained computer program for D -scoring, equating and item analysis, DELTA (Atanasov & Dimitrov, 2018).…”

“…In addition, as shown by Dimitrov and Luo (2018), the DSM for binary items can be readily applied with ordered polytomous items and the results parallel their counterparts obtained under the graded response model (GRM) in item response theory (IRT; Samejima, 1969, 1996). The measurement and practical efficiency of the DSM was also shown in the framework of multistage testing (Al-Mashary & Dimitrov, 2018; Han et al, 2018).…”

About the Equivalence of the Latent D-Scoring Model and the Two-Parameter Logistic Item Response Model

“…In Section 3, we show that identification constraints are needed for the estimation of the model. The latent D-scoring model is applied in psychometric areas of linking and equating [17], differential item functioning [15], and the development of multi-stage tests [18].…”