Investigation of Item Selection Methods According to Test Termination Rules in CAT Applications

Abstract

In this research, computerized adaptive testing item selection methods were investigated in regard to ability estimation methods and test termination rules. For this purpose, an item pool including 250 items and 2000 people were simulated (M = 0, SD = 1). A total of thirty computerized adaptive testing (CAT) conditions were created according to item selection methods (Maximum Fisher Information, a-stratification, Likelihood Weight Information Criterion, Gradual Information Ratio, and Kullback-Leibler), ability estimation methods (Maximum Likelihood Estimation, Expected a Posteriori Distribution), and test termination rules (40 items, SE < .20 and SE < .40). According to the fixed test-length stopping rule, the SE values that were obtained by using the Maximum Likelihood Estimation method were found to be higher than the SE values that were obtained by using the Expected a Posteriori Distribution ability estimation method. When ability estimation was Maximum Likelihood, the highest SE value was obtained from a-stratification item selection method when the test length is smaller then 30. Whereas, Kullback-Leibler item selection method yielded the highest SE value when the test length is larger then 30. According to Expected a Posteriori ability estimation method, the highest SE value was obtained from a-stratification item selection method in all test lengths. In the conditions where test termination rule was SE < .20, and Maximum Likelihood Ability Estimation method was used, the lowest and highest average number of items were obtained from the Gradual Information Ratio and Maximum Fisher Information item selection method, respectively. Furthermore, when the SE is lower than .20 and Expected a Posteriori ability estimation method was utilized, the lowest average number of items was obtained through Kullback-Leibler, and the highest was obtained through Likelihood Weight Information Criterion item selection method. In the conditions where the test termination rule was SE < .40, and ability estimation method was Maximum Likelihood Estimation, the maximum and minimum number of items were obtained by using Maximum Fisher Information and Kullback-Leibler item selection methods respectively. Additionally, when Expected a Posteriori ability estimation was used, the maximum and minimum number of items were obtained via Maximum Fisher Information and a-stratification item selection methods. For the cases where the stopping rule was SE < .20 and SE < .40 and Maximum Likelihood Estimation method was used, the average number of items were found to be highest in all item selection methods.

Investigation of Measurement Precision and Test Lengths in Computerized Adaptive Tests in Different Conditions

“…When the literature is examined, there are studies in which many aspects of CAT (content balancing, item pool properties, test length, etc.) are compared under different conditions (Boyd, 2003;Eroğlu & Kelecioğlu, 2012;Demir, 2018;Aybek & Çıkrıkçı, 2018;Sulak & Kelecioğlu, 2019;Kara, 2019). It is considered that the research will contribute to the field in terms of examining the measurement accuracy and test length when ability estimation methods, content balancing approaches, item exposure rates and termination rules are changed in CAT applications.…”

“…According to this method, an individual's ability level depends on a certain standard error and if a certain measurement precision is reached, the test is terminated (Demir, 2018).As a standard error-based termination rule, 0.30 and 0.40 cutting scores were used. These values were frequently studied in the literature and were critical values in terms of test termination rules to obtain a measurement precision (Aybek & Çıkrıkçı, 2018;Sulak & Kelecioğlu, 2019;Yao, 2012). For termination rules based on fixed test length, 15 and 30 items are preferred.…”

Bilgisayar Ortamında Bireye Uyarlanmış Test Uygulamalarında Ölçme Kesinliğinin ve Test Uzunluğunun Farklı Koşullar Altında İncelenmesi

“…BBT uygulamalarında, yüksek ayırt edicilik düzeyindeki maddeleri seçmeye eğilimli olan MFB madde seçme yönteminin testin başlangıcında yetenek kestiriminde yetersiz kaldığı belirtilmektedir (Deng, Ashley & Chang, 2010;Chang & Ying, 1999;Han, 2009;Veldkamp, 2012;Weissman, 2003). Böylelikle, KLB madde seçme yönteminin MFB yöntemine göre daha kesin yeterlik kestirimleri sağladığı sonucuna ulaşan araştırmalar (Eggen, 1999;Han, 2009;Reckase, 1983;Spray & Reckase, 1994;Sulak & Kelecioğlu, 2019) dikkate alınarak yöntemler karşılaştırılmalı olarak ele alınmıştır. Araştırmada, yetenek kestirim yöntemi olarak Maksimum Olabilirlik Kestirimi (MOK), Beklenen Sonsal Dağılım (BSD) ve Maksimum Sonsal Dağılım (MSD) yöntemleri ele alınmıştır.…”

Applicability and Efficiency of a Polytomous IRT-Based Computerized Adaptive Test for Measuring Psychological Traits

“…Test termination; The standard error rule (SE) is the most commonly used test termination rule (Babcock & Weiss, 2012). Considering the relationship between SE and measurement precision, .315, .385, and .500 SE are used, corresponding to measurement precision of .90, .85, and .75, respectively (Babcock & Weiss, 2012;Kezer, 2013;Sulak & Kelecioğlu, 2019).…”