The Use of Smartphone Keystroke Dynamics to Passively Monitor Upper Limb and Cognitive Function in Multiple Sclerosis: Longitudinal Analysis

Abstract

Background Typing on smartphones, which has become a near daily activity, requires both upper limb and cognitive function. Analysis of keyboard interactions during regular typing, that is, keystroke dynamics, could therefore potentially be utilized for passive and continuous monitoring of function in patients with multiple sclerosis. Objective To determine whether passively acquired smartphone keystroke dynamics correspond to multiple sclerosis outcomes, we investigated the association between keystroke dynamics and clinical outcomes (upper limb and cognitive function). This association was investigated longitudinally in order to study within-patient changes independently of between-patient differences. Methods During a 1-year follow-up, arm function and information processing speed were assessed every 3 months in 102 patients with multiple sclerosis with the Nine-Hole Peg Test and Symbol Digit Modalities Test, respectively. Keystroke-dynamics data were continuously obtained from regular typing on the participants’ own smartphones. Press-and-release latency of the alphanumeric keys constituted the fine motor score cluster, while latency of the punctuation and backspace keys constituted the cognition score cluster. The association over time between keystroke clusters and the corresponding clinical outcomes was assessed with linear mixed models with subjects as random intercepts. By centering around the mean and calculating deviation scores within subjects, between-subject and within-subject effects were distinguished. Results Mean (SD) scores for the fine motor score cluster and cognition score cluster were 0.43 (0.16) and 0.94 (0.41) seconds, respectively. The fine motor score cluster was significantly associated with the Nine-Hole Peg Test: between-subject β was 15.9 (95% CI 12.2-19.6) and within-subject β was 6.9 (95% CI 2.0-11.9). The cognition score cluster was significantly associated with the Symbol Digit Modalities Test between subjects (between-subject β –11.2, 95% CI –17.3 to –5.2) but not within subjects (within-subject β –0.4, 95% CI –5.6 to 4.9). Conclusions Smartphone keystroke dynamics were longitudinally associated with multiple sclerosis outcomes. Worse arm function corresponded with longer latency in typing both across and within patients. Worse processing speed corresponded with higher latency in using punctuation and backspace keys across subjects. Hence, keystroke dynamics are a potential digital biomarker for remote monitoring and predicting clinical outcomes in patients with multiple sclerosis. Trial Registration Netherlands Trial Register NTR7268; https://trialsearch.who.int/Trial2.aspx?TrialID=NTR7268

Disease Severity Classification using Passively Collected Smartphone-Based Keystroke Dynamics within Multiple Sclerosis

“…classification pipeline, as also shown in Figure 4. In Lam et al 13 It was considered a 28-day (clinical visit ±14 days) and 14-day (clinical visit ±7 days) aggregation period for the fine motor and cognitive clusters, respectively, under the assumption that fine motor and cognitive functions are stable within such time windows. Furthermore, a keystroke event count threshold of 50 events was used to remove days with insufficient data.…”

“…Finally, composite scores are created by averaging a cluster of features into single scores to reduce potential information overload 13 . More precisely, two Fine Motor Composite Score (FMCS), and a Cognition Composite Score (CCS) are derived, based on the hypothesis that timing-related features (PPL n , RRL n , HT n , and FT n ) are more related to fine motor skills, while error-related (PreCS n , and PostCS n ) and paralinguistic (APP n ) features are more related to cognitive processes.…”

“…After several preprocessing steps, the keystroke sequences are aggregated on a daily level using summary statistics such as mean and standard deviation (see Table 3 for a complete list). Finally, composite scores are created by averaging subsets of keystroke features tuned based on both a hypothesis and data-driven approach 13 .…”

Disease severity classification using passively collected smartphone-based keystroke dynamics within multiple sclerosis

“…Finally, composite scores are created by averaging a cluster of features into single scores to reduce potential information overload 13 . More precisely, two Fine Motor Composite Score (FMCS), and a Cognition Composite Score (CCS) are derived, based on the hypothesis that timing-related features ( , , , and ) are more related to fine motor skills, while error-related ( , and ) and paralinguistic ( ) features are more related to cognitive processes.…”

“…Additionally, KD have demonstrated a higher sensitivity to changes in disease activity, fatigue, and clinical disability compared to commonly used clinical measures via detection of important changes beyond measurement error on a group level 12 . Finally, the association between KD and clinical outcomes in longitudinal settings has been shown, namely that worse arm function corresponds with longer latencies in typing across and within patients, and worse processing speed corresponds with higher latencies relating to punctuations and backspaces across subjects 13 .…”