Reliability and Validity of Ambulatory Cognitive Assessments

Abstract

Mobile technologies are increasingly used to measure cognitive function outside of traditional clinic and laboratory settings. Although ambulatory assessments of cognitive function conducted in people’s natural environments offer potential advantages over traditional assessment approaches, the psychometrics of cognitive assessment procedures have been understudied. We evaluated the reliability and construct validity of ambulatory assessments of working memory and perceptual speed administered via smartphones as part of an ecological momentary assessment (EMA) protocol in a diverse adult sample (N=219). Results indicated excellent between-person reliability (≥.97) for average scores, and evidence of reliable within-person variability across measurement occasions (.41–.53). The ambulatory tasks also exhibited construct validity, as evidence by their loadings on working memory and perceptual speed factors defined by the in-lab assessments. Our findings demonstrate that averaging across brief cognitive assessments made in uncontrolled naturalistic settings provide measurements that are comparable in reliability to assessments made in controlled laboratory environments.

Applications for self‐administered mobile cognitive assessments in clinical research: A systematic review

“…In general, studies conducted prior to 2009 used personal digital assistants (PDAs; n = 4) and studies after 2009 used a mobile phone ( n = 8). Four studies specified if the mobile phone was a smartphone (Brouillette et al, ; Schweitzer et al, ; Sliwinski et al, ; Timmers et al, ), while the other studies using mobile phones either did not use smartphones or did not make this specification. In all but one study (Brouillette et al, ) the mobile devices were provided to the participants.…”

A systematic review of reviews of neurocognitive functioning in eating disorders: The state‐of‐the‐literature and future directions

“…Emerging ecological momentary assessment (EMA) research suggests some domains such as inhibitory control vary within persons without EDs (e.g., Powell, McMinn, & Allan, 2017), and thus could be state-dependent, and recent progress has been made in adapting cognitive tasks for mobile assessment (e.g., Moore, Swendsen, & Depp, 2017; Sliwinski et al, 2018). In EDs, it is yet unclear to what extent these neurocognitive abilities fluctuate within persons and how state neurocognitive deficits may maintain ED behaviors.…”

“…Thus, longitudinal research is needed to clarify whether specific neurocognitive deficits reflect endophenotypes that begin prior to the illness and persist after recovery, start with the onset of the illness and improve with symptom reduction, or might be best viewed as scars of the illness that begin with the onset of the illness, but persist after recovery. Given the methodological difficulties associated with longitudinal studies, one promising option to address such obstacles going forward is by harnessing the advantages of ambulatory and web-based neurocognitive assessment methods that are more easily accessible to participants (e.g., Hansen, Haferstrom, Brunner, Lehn, & Håberg, 2015; Morrison, Simone, Ng, & Hardy, 2015; Sliwinski et al, 2018). …”

Cognitive performance of breast cancer survivors in daily life: Role of fatigue and depressed mood

“…Prompts occurred around the hours of 9 AM , 12 PM , 3 PM , 6 PM , and 9 PM , but timing appeared quasi‐random to participants so that they would not anticipate them. A duration of 14 days was selected for the assessment period based upon previous studies that indicated assessing cognitive performance for at least 10 days allowed practice effects to be disentangled from cognitive performance . At the end of the EMA phase, participants were contacted to instruct them how to return their smartphones via the postage paid mailer.…”