Ecological Momentary Assessment of Cognition in Clinical and Community Samples: Reliability and Validity Study

Abstract

Background The current methods of evaluating cognitive functioning typically rely on a single time point to assess and characterize an individual’s performance. However, cognitive functioning fluctuates within individuals over time in relation to environmental, psychological, and physiological contexts. This limits the generalizability and diagnostic utility of single time point assessments, particularly among individuals who may exhibit large variations in cognition depending on physiological or psychological context (eg, those with type 1 diabetes [T1D], who may have fluctuating glucose concentrations throughout the day). Objective We aimed to report the reliability and validity of cognitive ecological momentary assessment (EMA) as a method for understanding between-person differences and capturing within-person variation in cognition over time in a community sample and sample of adults with T1D. Methods Cognitive performance was measured 3 times a day for 15 days in the sample of adults with T1D (n=198, recruited through endocrinology clinics) and for 10 days in the community sample (n=128, recruited from TestMyBrain, a web-based citizen science platform) using ultrabrief cognitive tests developed for cognitive EMA. Our cognitive EMA platform allowed for remote, automated assessment in participants’ natural environments, enabling the measurement of within-person cognitive variation without the burden of repeated laboratory or clinic visits. This allowed us to evaluate reliability and validity in samples that differed in their expected degree of cognitive variability as well as the method of recruitment. Results The results demonstrate excellent between-person reliability (ranging from 0.95 to 0.99) and construct validity of cognitive EMA in both the sample of adults with T1D and community sample. Within-person reliability in both samples (ranging from 0.20 to 0.80) was comparable with that observed in previous studies in healthy older adults. As expected, the full-length baseline and EMA versions of TestMyBrain tests correlated highly with one another and loaded together on the expected cognitive domains when using exploratory factor analysis. Interruptions had higher negative impacts on accuracy-based outcomes (β=−.34 to −.26; all P values <.001) than on reaction time–based outcomes (β=−.07 to −.02; P<.001 to P=.40). Conclusions We demonstrated that ultrabrief mobile assessments are both reliable and valid across 2 very different clinic versus community samples, despite the conditions in which cognitive EMAs are administered, which are often associated with more noise and variability. The psychometric characteristics described here should be leveraged appropriately depending on the goals of the cognitive assessment (eg, diagnostic vs everyday functioning) and the population being studied.

Remote and unsupervised digital memory assessments can reliably detect cognitive impairment in Alzheimer’s disease

“…This is particularly true for unsupervised and remote digital assessments that are completed in an unstandardized environment of the participant’s choice. Recent work using smartphone-based cognitive assessments in samples of older participants could successfully demonstrate high construct validity across various cognitive domains (Nicosia et al, 2022b; Öhman et al, 2022; Papp et al, 2021; Singh et al, 2023; Vyshedskiy et al, 2022). Both the ORR and the MDT-OS are intended to measure memory.…”

Dynamic associations between glucose and ecological momentary cognition in Type 1 Diabetes

“…2a ) is a visuospatial task with demands on processing speed and short-term working memory 65 . It was adapted for remote administration from the Wechsler Adult Intelligence Scale digit symbol coding task 66 , 67 and has been previously validated for cognitive EMA in T1D 68 . During administration, participants were presented with both a target and a digit-symbol pairing key (Fig.…”

“…2b ) is an executive functioning task that requires sustained attention, cognitive control, and response inhibition 69 , 70 . It has been previously validated for cognitive EMA in T1D 68 . In each trial, participants viewed a circular, grayscale image of a city or mountain (Fig.…”

Aberrant positive affect dynamics in individuals with subthreshold depression: Evidence from laboratory and real-world assessments

“…Participants from Sample 1 were included in the EMA to maintain consistency of results. Furthermore, robust ecological validity and statistical power require support from a larger sample size ( Singh et al., 2023 ), and therefore, we increased the original sample size. Fifty-five additional participants were recruited using the same method.…”