Using response time data to reduce testing time in cognitive tests.

Abstract

Achievement estimates are often based on either number correct scores or IRT-based ability parameters. Van der Linden (2007) and other researchers (e.g., Fox, Klein Entink, & van der Linden, 2007; Ranger, 2013) have developed psychometric models that allow for joint estimation of speed and item parameters using both response times and response data. This paper presents an application of this type of approach to a battery of 4 types of fluid reasoning measures, administered to a large sample of a highly educated examinees. We investigate the extent to which incorporation of response times in ability estimates can be used to inform the potential development of shorter test forms. In addition to exploratory analyses and response time data visualizations, we specifically consider the increase in precision of ability estimates given the addition of response time data relative to use of item responses alone. Our findings indicate that there may be instances where test forms can be substantially shortened without any reduction in score reliability, when response time information is incorporated into the item response model. (PsycINFO Database Record

Joint modeling of the two-alternative multidimensional forced-choice personality measurement and its response time by a Thurstonian D-diffusion item response model

“…The first is "to obtain more accurate proficiency level estimates," and the second is "to estimate examinee performance on a separate latent trait." Although our situation may be different in that the current study considers personality measurement while Bertling and Weeks (2018) considered ability measurement, we believe that the proposed model is closer to the latter case. That is, the proposed Thurstonian D-diffusion IRT model estimates the item and respondent parameters on the basis of psychological theory models of comparative judgment, the diffusion process, and the distance-difficulty relationship.…”

“…In computer-based measurement, the response time information can be obtained with little additional cost along with the item responses. Various studies have discussed the utility of incorporating response time information in, e.g., classifying response behaviors, removing aberrant responses, and improving the estimation accuracy (e.g., Bertling and Weeks, 2018;Kong, Wise, & Bhola, 2007;Wang, Xu, & Shang, 2018;Wise & Kong, 2005). The general validity of online measurement has been demonstrated for both personality characteristics and response times (e.g., Condon, 2018;Raz, Bar-Haim, Sadeh, & Dan, 2014).…”

Effects on Test Performance and Efficiency of Technology-Enhanced Items: An Analysis of Drag-and-Drop Response Interactions

“…These efficiency gains were obtained by innovating on the computer interfaces used to implement the test items. Other studies have explored efficiency gains by reducing the number of items in a test (Barnard, 2018; Bertling & Weeks, 2018).…”

“…Few studies have investigated the properties of technology-enhanced test items—so it is difficult for test designers to decide whether to include them in tests given the extra costs, complexities, and risks involved (Qian et al., 2018; Russell & Moncaleano, 2019). Some key research questions include how technology-enhanced items compare with other item types in terms of response accuracy, reliability and discrimination; and whether technology-enhanced items affect the time needed to respond compared with conventional items (Bertling & Weeks, 2018; Bryant, 2017; De Boeck et al., 2017).…”

On the Speed Sensitivity Parameter in the Lognormal Model for Response Times and Implications for High-Stakes Measurement Practice

“…3.This may be caused by the labeling of the inverse of the residual variance as a discrimination parameter, which is usually a term used for slope parameters. For example, Bertling and Weeks (2018) cite van der Linden (2006) but introduce the model with αk as a slope parameter instead of the inverse of the residual variance.…”