Reliability of a Smartphone Neuropsychological Performance Self-Monitoring App Named Roberto: Correlational Study (Preprint)

Abstract

BACKGROUND As technology advances to create new devices capable of neuropsychological measurements, new possibilities emerge. Smartphones are one such device that many people own. Smartphone-based measurements may increase consumer, clinician, and researcher access to neuropsychological data. To explore smartphones for this purpose, it is important to determine their psychometric rigor. OBJECTIVE Due to convenience and accessibility, smartphone measurements may complement and surpass computerized neuropsychological and paper-and-pencil measurements but lack psychometric evaluation. We analyzed test-retest reliability of a neuropsychological self-monitoring app named Roberto. METHODS Participants downloaded and self-administered Roberto, unsupervised and without assigned location. Roberto scoring used a multiplicative approach derived from general systems performance theory (GSPT) and the elemental resource model (ERM) for human performance. Various modules measured neuromotor and cognitive performance during a six-minute Roberto administration session. We analyzed data from users who completed their first two sessions within 14 days (n = 69). RESULTS Analyses included Pearson’s product-moment correlations for module scores to assess reliability, a paired samples t-test (t = -3.087, p = 0.003) on composite scores for sessions 1 and 2 to assess practice effects, and a repeated-measures MANOVA using session number as an independent variable and module scores as a dependent variable to determine if module scores differed between sessions 1 and 2, followed by univariate ANOVAs to determine which modules differed. Reliability coefficients ranged from 0.296 to 0.725 (p values less than 0.05); most coefficients were above 0.53. We found practice effects; scores on session 2 were 46% higher than on session 1, and MANOVA revealed significant differences between sessions, and univariate ANOVAs show that three modules had significantly different scores on session 2 than session 1. CONCLUSIONS Results demonstrate feasibility of a smartphone app used in ecologically robust ways, even in suboptimal conditions. Future research needs to evaluate reliability between test environments. Additional studies need to investigate differences between smartphone type and specifications, such as screen size and operating system. Furthermore, our study found practice effects over multiple Roberto administrations; this introduces the need to investigate the dynamics of these effects on smartphones. Critically, future work needs to investigate validity of neuropsychological data obtained from short measurement sessions compared to full neuropsychological exams.