Using Response-Time Constraints to Control for Differential Speededness in Computerized Adaptive Testing

Linden, Willem J. van der; Scrams, David J.; Schnipke, Deborah L.

doi:10.1177/01466219922031329

Cited by 103 publications

(113 citation statements)

References 15 publications

Supporting

Mentioning

106

Contrasting

Order By: Relevance

“…These include assessing examinee motivation levels, particularly for low stakes tests [132][133][134], evaluating strategy use, for example by differences in response times for subgroups employing different problem solving strategies [130,135], or even the same individual employing different strategies at different points in the test [136], and evaluating cultural differences in pacing and time management during test taking [137]. Other applications include detecting cheating [97], assembling parallel forms [138], and item selection in adaptive testing [139,140], which may be particularly important for ensuring score comparability 16 across sets of items that might be similar in difficulty but differ in their time intensity [142,143]. Finally, although the applications here focused on response time for cognitive tests, another application only briefly mentioned here (Section 6.4) is to model response time on personality and attitude assessments [31,144,145].…”

Section: Other Uses Of Response Timementioning

confidence: 99%

Use of Response Time for Measuring Cognitive Ability

Kyllonen

2016

J. Intell.

109

View full text Add to dashboard Cite

Abstract:The purpose of this paper is to review some of the key literature on response time as it has played a role in cognitive ability measurement, providing a historical perspective as well as covering current research. We discuss the speed-level distinction, dimensions of speed and level in cognitive abilities frameworks, speed-accuracy tradeoff, approaches to addressing speed-accuracy tradeoff, analysis methods, particularly item response theory-based, response time models from cognitive psychology (ex-Gaussian function, and the diffusion model), and other uses of response time in testing besides ability measurement. We discuss several new methods that can be used to provide greater insight into the speed and level aspects of cognitive ability and speed-accuracy tradeoff decisions. These include item-level time limits, the use of feedback (e.g., CUSUMs), explicit scoring rules that combine speed and accuracy information (e.g., count down timing), and cognitive psychology models. We also review some of the key psychometric advances in modeling speed and level, which combine speed and ability measurement, address speed-accuracy tradeoff, allow for distinctions between response times on items responded to correctly and incorrectly, and integrate psychometrics with information-processing modeling. We suggest that the application of these models and tools is likely to advance both the science and measurement of human abilities for theory and applications.

show abstract

Section: Other Uses Of Response Timementioning

confidence: 99%

Use of Response Time for Measuring Cognitive Ability

Kyllonen

2016

J. Intell.

109

View full text Add to dashboard Cite

show abstract

“…However, previous results obtained for the between-person level are mixed, possibly depending as well on analysis method and task content. In fact, positive RTACs were found for reasoning [6,10] and problem solving [1], null RTACs for arithmetic tasks [11], and negative RTACs for basic computer operation skills [12] and reading tasks [1].…”

Section: Item Response Time and Item Successmentioning

confidence: 99%

Sometimes More Is Better, and Sometimes Less Is Better: Task Complexity Moderates the Response Time Accuracy Correlation

et al. 2016

View full text Add to dashboard Cite

This study addresses the relationship between item response time and item accuracy (i.e., the response time accuracy correlation, RTAC) in figural matrices tests. The dual processing account of response time effects predicts negative RTACs in tasks that allow for relatively automatic processing and positive RTACs in tasks that require controlled processing. Contrary to these predictions, several studies found negative RTACs for reasoning tests. Nevertheless, it was demonstrated that the RTAC is moderated by task complexity (i.e., the interaction between person ability and item difficulty) and that under conditions of high complexity (i.e., low ability and high difficulty) the RTAC was even slightly positive. The goal of this study was to demonstrate that with respect to task complexity the direction of the RTAC (positive vs. negative) can change substantially even within a single task paradigm (i.e., figural matrices). These predictions were tested using a figural matrices test that employs a constructed response format and has a broad range of item difficulties in a sample with a broad range of ability. Confirming predictions, strongly negative RTACs were observed when task complexity was low (i.e., fast responses tended to be correct). With increasing task complexity, the RTAC flipped to be strongly positive (i.e., slow responses tended to be correct). This flip occurred earlier for people with lower ability, and later for people with higher ability. Cognitive load of the items is suggested as an explanation for this phenomenon.

show abstract

“…The discrimination parameters with respect to speed (i.e., a) were generated from the U(1, 3) distribution whereas the time-intensity parameters were generated in two different ways: (i) assuming no correlation with IRT b parameters, the b parameters were generated from the U(3, 5) distribution (van der Linden, 2008) and (ii) assuming a 0.65 correlation with IRT b parameters (van der Linden et al, 1999), the b parameters were generated by sampling a separate value from its conditional distribution given b; that is, Five hundred examinees were simulated at each of 25 evenly spaced true ability values from À3:0 to + 3:0 with an increment of 0.25. Similar to the time-intensity parameters, examinees' speed parameters were also generated in two different ways: (i) assuming no correlation with u, the t parameters were generated from the N(0, 0.24 2 ) distribution and (ii) assuming a 0.59 correlation with u (van der Linden, 1999), the t parameters were generated by sampling a separate value from its conditional distribution given u; that is, N (m t + r ut s t =s u (u À m u ), s For approach (ii), m t = 0 and s 2 t = 0:24 2 were used to give the same mean and variance of the t parameters as in approach (i).…”

Section: Designmentioning

confidence: 99%

Utilizing Response Times in Computerized Classification Testing

Sie

Finkelman

Riley

et al. 2015

Applied Psychological Measurement

View full text Add to dashboard Cite

Utilizing response times in computerized classification testingSie, H.; Finkelman, M.D.; Riley, B.; Smits, N. Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library: http://uba.uva.nl/en/contact, or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible. Abstract A well-known approach in computerized mastery testing is to combine the Sequential Probability Ratio Test (SPRT) stopping rule with item selection to maximize Fisher information at the mastery threshold. This article proposes a new approach in which a time limit is defined for the test and examinees' response times are considered in both item selection and test termination. Item selection is performed by maximizing Fisher information per time unit, rather than Fisher information itself. The test is terminated once the SPRT makes a classification decision, the time limit is exceeded, or there is no remaining item that has a high enough probability of being answered before the time limit. In a simulation study, the new procedure showed a substantial reduction in average testing time while slightly improving classification accuracy compared with the original method. In addition, the new procedure reduced the percentage of examinees who exceeded the time limit.

show abstract

Using Response-Time Constraints to Control for Differential Speededness in Computerized Adaptive Testing

Cited by 103 publications

References 15 publications

Use of Response Time for Measuring Cognitive Ability

Use of Response Time for Measuring Cognitive Ability

Sometimes More Is Better, and Sometimes Less Is Better: Task Complexity Moderates the Response Time Accuracy Correlation

Utilizing Response Times in Computerized Classification Testing

Contact Info

Product

Resources

About