Investigation of Classification Accuracy, Test Length and Measurement Precision at Computerized Adaptive Classification Tests

Abstract

This study aims to compare Sequential Probability Ratio Test (SPRT) and Confidence Interval (CI) classification criteria, Maximum Fisher Information method on the basis of estimated-ability (MFI-EB) and Cut-Point (MFI-CB) item selection methods while ability estimation method is Weighted Likelihood Estimation (WLE) in Computerized Adaptive Classification Testing (CACT), according to the Average Classification Accuracy (ACA), Average Test Length (ATL), and measurement precision under content balancing (Constrained Computerized Adaptive Testing: CCAT and Modified Multinomial Model: MMM) and item exposure control (Sympson-Hetter Method: SH and Item Eligibility Method: IE) when the classification is done based on two, three, or four categories for a unidimensional pool of dichotomous items. Forty-eight conditions are created in Monte Carlo (MC) simulation for the data, generated in R software, including 500 items and 5000 examinees, and the results are calculated over 30 replications. As a result of the study, it was observed that CI performs better in terms of ATL, and SPRT performs better in ACA and correlation, bias, Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE) values, sequentially; MFI-EB is more useful than MFI-CB. It was also seen that MMM is more successful in content balancing, whereas CCAT is better in terms of test efficiency (ATL and ACA), and IE is superior in terms of item exposure control though SH is more beneficial in test efficiency. Besides, increasing the number of classification categories increases ATL but decreases ACA, and it gives better results in terms of the correlation, bias, RMSE, and MAE values.

The Effect of Strong and Weak Unidimensional Item Pools on Computerized Adaptive Classification Testing

“…When CAT and CACT studies are examined, it is seen that Monte Carlo (MC) and Post Hoc (PH) simulations are often carried out in the R environment (Ayan, 2018;Demir, 2019;Erdem Kara, 2019;Gündeğer, 2017;Özdemir, 2015). Some studies present the dimensionality of the item pools (e.g., Ayan, 2018;Aybek, 2016;Demir, 2019;Erdem Kara, 2019;Gündeğer, 2017;Özdemir, 2015;Şahin, 2017); some present the information of unidimensionality with the item loads (e.g., Ayan, 2018;Doğruöz, 2018;Gündeğer, 2017;Şenel, 2017); and some present only the number of items as item pool characteristic (Kaçar, 2016). In fact, when MC data are generated by the software based on unidimensionality, it is important to show some evidences about how the items represent the latent trait.…”