Proposal of a Wearable Multimodal Sensing-Based Serious Games Approach for Hand Movement Training After Stroke

Abstract

Stroke often leads to hand motor dysfunction, and effective rehabilitation requires keeping patients engaged and motivated. Among the existing automated rehabilitation approaches, data glove-based systems are not easy to wear for patients due to spasticity, and single sensor-based approaches generally provided prohibitively limited information. We thus propose a wearable multimodal serious games approach for hand movement training after stroke. A force myography (FMG), electromyography (EMG), and inertial measurement unit (IMU)-based multi-sensor fusion model was proposed for hand movement classification, which was worn on the user’s affected arm. Two movement recognition-based serious games were developed for hand movement and cognition training. Ten stroke patients with mild to moderate motor impairments (Brunnstrom Stage for Hand II-VI) performed experiments while playing interactive serious games requiring 12 activities-of-daily-living (ADLs) hand movements taken from the Fugl Meyer Assessment. Feasibility was evaluated by movement classification accuracy and qualitative patient questionnaires. The offline classification accuracy using combined FMG-EMG-IMU was 81.0% for the 12 movements, which was significantly higher than any single sensing modality; only EMG, only FMG, and only IMU were 69.6, 63.2, and 47.8%, respectively. Patients reported that they were more enthusiastic about hand movement training while playing the serious games as compared to conventional methods and strongly agreed that they subjectively felt that the proposed training could be beneficial for improving upper limb motor function. These results showed that multimodal-sensor fusion improved hand gesture classification accuracy for stroke patients and demonstrated the potential of this proposed approach to be used as upper limb movement training after stroke.

Behavior Change Approaches in Digital Technology–Based Physical Rehabilitation Interventions Following Stroke: Scoping Review

“…An overview of study intervention characteristics is provided ( Table 1 ). Interventions were focused on upper limb rehabilitation in almost two-thirds (65/103, 63%) of the studies [ 46 - 49 , 51 , 54 - 59 , 62 - 65 , 68 , 71 , 72 , 74 , 75 , 77 - 81 , 85 - 88 , 92 , 95 , 96 , 99 , 101 - 103 , 105 - 108 , 110 , 112 , 113 , 116 - 118 , 121 , 123 - 125 , 127 , 128 , 132 , 133 , 135 - 137 , 139 - 142 , 144 - 147 ]. Nearly all interventions (96/103, 93%) [ 46 - 80 , 84 - 94 , 96 - 117 , 119 - 121 , 124 - 148 ] were delivered to individual participants, with over half (62/103, 60%) [ 46 - 50 , 53 - 58 , 60 , 61 , 64 - 70 , 72 , 74 -…”

“…Of the 103 studies, over half (n=57, 55%) of the studies [ 46 , 47 , 51 - 54 , 57 , 61 , 63 , 67 , 68 , 70 , 71 , 73 , 75 - 78 , 81 , 84 - 86 , 88 - 91 , 93 , 95 , 96 , 98 , 100 , 102 - 104 , 106 , 109 , 112 , 114 , 115 , 123 - 126 , 129 , 130 , 132 , 133 , 135 - 138 , 140 , 143 - 147 ] included 1 type of DHT, 30 (29%) studies [ 48 , 49 , 55 , 56 , 58 - 60 , 62 , 64 , 69 , 83 , 92 , 94 , 97 , 99 , 101 , 105 , 107 , 108 , 110 , 111 , 113 , 116 , 118 , 121 , …”

Multimodal Sensing in Stroke Motor Rehabilitation

“…For example, in gesture recognition, the study of Song has shown that the classification accuracy of the combination of FMG‐EMG‐IMU (81%) is significantly higher than any single sensing mode (EMG, 69.6%, FMG, 63.2%, IMU, 47.8%). [ 207 ] The detection of motion intention by the sensor is helpful for human‐computer interaction. Park introduced elastic tension sensors and pressure sensors into the hand orthotics to assist EMG in identifying the motion intention of patients, thus achieving accurate assistance in grasping training (global accuracy = 86%).…”

Automatic Upper-Limb Brunnstrom Recovery Stage Evaluation via Daily Activity Monitoring

“…Mean Frequency [32], [33] MNF -Sample Entropy [26], [30] SampEn m = 2, r = 0.2 × σ Difference Absolute Standard Deviation Value [33], [34] DASDV -Difference of Maximum and Minimum Value [4], [35] DMMV -Energy [12], [26] ENE -Hjorth1 (Activity) or Variance [7], [36] Hjorth1 (or VAR) -Interquartile Range [7], [26] IQR [26] Kurt -Log Detector [27], [34] LD -Standard Deviation Value [5], [7] SD -Skewness [7], [18] Skew -Linear Prediction Coefficient 2 [16], [37] 2nd LPC -Linear Prediction Coefficient 3 [16], [37] 3rd LPC -Zero Crossing [7], [38] ZC threshold : 0.03 × σrest Slope Sign Change [7], [38] SSC threshold : 0.03 × σrest Spectral Entropy [26] SpEn -Simple Square Integral [27], [33] SSI -Waveform Length [18], [27], [28] WL -Auto-Regressive Coefficient 1 [39], [40] AR1 order : 4 Auto-Regressive Coefficient 2 [39], [40] AR2 order : 4 Auto-Regressive Coefficient 3 [39], [40] AR3 order : 4 Auto-Regressive Coefficient 4 [39], [40] AR4 order : 4 Maximum-to-M...…”