Inferring Cognitive Abilities from Response Times to Web-Administered Survey Items in a Population-Representative Sample

Abstract

Monitoring of cognitive abilities in large-scale survey research is receiving increasing attention. Conventional cognitive testing, however, is often impractical on a population level highlighting the need for alternative means of cognitive assessment. We evaluated whether response times (RTs) to online survey items could be useful to infer cognitive abilities. We analyzed >5 million survey item RTs from >6000 individuals administered over 6.5 years in an internet panel together with cognitive tests (numerical reasoning, verbal reasoning, task switching/inhibitory control). We derived measures of mean RT and intraindividual RT variability from a multilevel location-scale model as well as an expanded version that separated intraindividual RT variability into systematic RT adjustments (variation of RTs with item time intensities) and residual intraindividual RT variability (residual error in RTs). RT measures from the location-scale model showed weak associations with cognitive test scores. However, RT measures from the expanded model explained 22–26% of the variance in cognitive scores and had prospective associations with cognitive assessments over lag-periods of at least 6.5 years (mean RTs), 4.5 years (systematic RT adjustments) and 1 year (residual RT variability). Our findings suggest that RTs in online surveys may be useful for gaining information about cognitive abilities in large-scale survey research.

Digital questionnaire response time (DQRT): a ubiquitous and low-cost digital assay of cognitive status

“…To a lesser extent, DQRT was related to working memory, but not model-based planning. These results are in line with previous studies 15,17,18 and support a cognitive characterization of DQRT as a measure of 'cognitive processing speed'. Our findings were consistent across samples and questionnaire sets, including common Likertscale multiple-choice items and brief EMA-style slider items.…”

“…Digital questionnaire response time (DQRT) is an increasingly common form of research paradata that some have suggested can serve as a proxy for cognitive functioning 14,15,17 . Here, we use a large smartphone-based dataset, with crosssectional and longitudinal data from questionnaires, experience sampling, and three cognitive tasks, to provide a comprehensive validation of DQRT.…”

“…Crucially, recent research has suggested these response times may also serve as a marker of individual differences in cognitive ability itself 15 .…”

“…In perhaps the most comprehensive investigation of this to date, the association between survey response times and cognitive test scores was assessed crosssectionally as well as longitudinally across lags of up to 6.5 years in over 6000 individuals from an internet panel 15 . This study found that, in terms of cognition, survey response times mapped individual differences in task switching, and to a lesser extent, verbal and quantitative reasoning.…”

Can you tell people’s cognitive ability level from their response patterns in questionnaires?

“…Self-report surveys are increasingly administered electronically over the internet, and the results should be replicated with responses from web-administered surveys. Web-based data collection also provides access to additional sources of survey response behaviors that were not considered here, including item response latencies recorded passively as paradata alongside the actual item responses, which have previously proven useful as indicators of people's cognitive ability (Junghaenel et al, 2023;Schneider, Junghaenel et al, 2023b).…”