Utilizing Response Time Distributions for Item Selection in CAT

Abstract

Traditional methods for item selection in computerized adaptive testing only focus on item information without taking into consideration the time required to answer an item. As a result, some examinees may receive a set of items that take a very long time to finish, and information is not accrued as efficiently as possible. The authors propose two item-selection criteria that utilize information from a lognormal model for response times. The first modifies the maximum information criterion to maximize information per time unit. The second is an inverse time-weighted version of a-stratification that takes advantage of the response time model, but achieves more balanced item exposure than the information-based techniques. Simulations are conducted to compare these procedures against their counterparts that ignore response times, and efficiency of estimation, time-required, and item exposure rates are assessed.

Utilizing Response Times in Computerized Classification Testing

“…Equation 6 for the distribution of T j (Fan et al, 2012). Using maximum likelihood to estimate both u and t, Fan et al showed that estimation accuracy only differed slightly between the original maximum information criterion in Equation 15 and that in Equation 17, but that using the latter substantially reduced the average time needed by examinees to complete the test.…”

“…Using maximum likelihood to estimate both u and t, Fan et al showed that estimation accuracy only differed slightly between the original maximum information criterion in Equation 15 and that in Equation 17, but that using the latter substantially reduced the average time needed by examinees to complete the test. For the current study, the ''maximum information per time unit'' framework introduced by Fan et al (2012) is extended to the current context of CCT. It is seen in Equation 17 that the numerator of the item selection criterion is evaluated at an interim ability estimateû kÀ1 , which, for CCT, results in higher ATL than Fisher information at the mastery threshold (Spray & Reckase, 1994).…”

“…Recently, Fan et al (2012) introduced a CAT item selection method that chose items having the ''maximum information per time unit.'' Specifically, they proposed choosing the kth test item to be…”

“…The current research aims at extending the work of Fan et al (2012) by developing a CCT procedure that is suited to tests with time restrictions. The procedure includes both an item selection method and a stopping rule that take the element of time into consideration.…”

“…Their applications in the framework of computerized adaptive testing (CAT) include detection of aberrant responses (van der Linden & van Krimpen-Stoop, 2003), detection of advanced item knowledge (Meijer & Sotaridona, 2006), control of speededness (van der Linden, 2009;van der Linden, Scrams, & Schnipke, 1999;van der Linden & Xiong, 2013), and improvement of item selection rules (Fan, Wang, Chang, & Douglas, 2012;van der Linden, 2008). In particular, the study by Fan et al (2012) showed that by taking into account examinees' expected RTs when choosing test items, the average time needed by examinees to complete the test can be substantially reduced, albeit with a small loss in estimation accuracy. The study, however, did not assess performance of the proposed item selection method in the context of CCT.…”