Improving precision of ability estimation: Getting more from response times

Abstract

By considering information about response time (RT) in addition to response accuracy (RA), joint models for RA and RT such as the hierarchical model can improve the precision with which ability is estimated over models that only consider RA. The hierarchical model, however, assumes that only the person's speed is informative of ability. This assumption of conditional independence between RT and ability given speed may be violated in practice, and ignores collateral information about ability that may be present in the residual RTs. We propose a posterior predictive check for evaluating the assumption of conditional independence between RT and ability given speed. Furthermore, we propose an extension of the hierarchical model that contains cross-loadings between ability and RT, which enables one to take additional collateral information about ability into account beyond what is possible in the standard hierarchical model. A Bayesian estimation procedure is proposed for the model. Using simulation studies, the performance of the model is evaluated in terms of parameter recovery, and the possible gain in precision over the standard hierarchical model and an RA-only model is considered. The model is applied to data from a high-stakes educational test.

Detection of Item Preknowledge Using Response Times

“…It is not known whether any RTM perfectly reflects reality or fits real data adequately. For example, even though the LNMRT is quite popular, researchers such as Bolsinova and Tijmstra (2018) and Ranger (2013) pointed to some limitations of the model. Therefore, to examine the properties of Λ s , simulations based on real data were used rather than simulations based on data generated from a RTM.…”

A General Three-Parameter Logistic Model With Time Effect

“…The collected response times provide a valuable source of information on examinees and test items. For example, response times can be used to improve the accuracy of ability estimates (van der Linden, 2007 ; Klein Entink et al, 2009a ; van der Linden and Glas, 2010 ; Wang et al, 2013 , 2018a ; Wang and Xu, 2015 ; Fox and Marianti, 2016 ; Bolsinova and Tijmstra, 2018 ; De Boeck and Jeon, 2019 ), to detect rapid guessing and cheating behavior (van der Linden and Guo, 2008 ; van der Linden, 2009 ; Wang and Xu, 2015 ; Pokropek, 2016 ; Qian et al, 2016 ; Skorupski and Wainer, 2017 ; Wang et al, 2018a , b ; Lu et al, 2019 ; Sinharay and Johnson, 2019 ; Zopluoglu, 2019 ), to evaluate the speededness of tests (Schnipke and Scrams, 1997 ; van der Linden et al, 2007 ), and to design more efficient tests (Bridgeman and Cline, 2004 ; Chang, 2004 ; Choe et al, 2018 ).…”

A New Person‐Fit Statistic for the Lognormal Model for Response Times

“…The LNMRT is arguably one of the most popular RTMs. The model was considered, either to analyze only the response times, or to analyze the response times and item scores, by, for example, Bolsinova and Tijmstra (), Boughton, Smith, and Ren (), Glas and van der Linden (), Qian, Staniewska, Reckase, and Woo (), van der Linden (, , ), van der Linden and Glas (), and van der Linden and Guo (). Bolsinova and Tijmstra (, p. 13) commented that the LNMRT is used in most applications of RTM.…”

“…The use of RTMs has been suggested to improve precision of examinee ability estimates (e.g., Bolsinova & Tijmstra, ), to detect test fraud (e.g., van der Linden & Guo, ), to detect speededness (e.g., Schnipke & Scrams, ), to improve test construction (e.g., van der Linden, ), and to test substantive theories about cognitive processes (e.g., van der Maas, Molenaar, Maris, Kievit, and Borsboom, ). Several RTMs have been suggested by, for example, Bolsinova and Tijmstra (), Entink, Fox, and van der Linden (), Entink, van der Linden, and Fox (), E. Maris (), G. Maris and van der Maas (), Rasch (), Schnipke and Scrams (), Thissen (), van der Linden (, ), van der Maas et al. (), and Wang and Hanson ().…”