No abstract

We are developing a system for long term Semi-Automated Rehabilitation At the Home (SARAH) that relies on low-cost and unobtrusive video-based sensing. We present a cyber-human methodology used by the SARAH system for automated assessment of upper extremity stroke rehabilitation at the home. We propose a hierarchical model for automatically segmenting stroke survivor's movements and generating training task performance assessment scores during rehabilitation. The hierarchical model fuses expert therapist knowledge-based approaches with data-driven techniques. The expert knowledge is more observable in the higher layers of the hierarchy (task and segment) and therefore more accessible to algorithms incorporating high level constraints relating to activity structure (i.e., type and order of segments per task). We utilize an HMM and a Decision Tree model to connect these high level priors to data driven analysis. The lower layers (RGB images and raw kinematics) need to be addressed primarily through data driven techniques. We use a transformer based architecture operating on low-level action features (tracking of individual body joints and objects) and a Multi-Stage Temporal Convolutional Network(MS-TCN) operating on raw RGB images. We develop a sequence combining these complimentary algorithms effectively, thus encoding the information from different layers of the movement hierarchy. Through this combination, we produce a robust segmentation and task assessment results on noisy, variable and limited data, which is characteristic of low cost video capture of rehabilitation at the home. Our proposed approach achieves 85% accuracy in per-frame labeling, 99% accuracy in segment classification and 93% accuracy in task completion assessment. Although the methodology proposed in this paper applies to upper extremity rehabilitation using the SARAH system, it can potentially be used, with minor alterations, to assist automation in many other movement rehabilitation contexts (i.e., lower extremity training for neurological accidents).

Telehealth rehabilitation systems aimed at providing physical and occupational therapy in the home face considerable challenges in terms of clinician and therapist buy-in, system and training costs and patient and caregiver acceptance. Understanding the optimal workflow process to support practitioners in delivering quality care in partnership with assistive technologies is significant. We describe the iterative co-development of our hybrid physical/digital workflow process for assisting therapists with the setup and cali bration of a computer vision based system for remote rehabilitation Through an interdisciplinary collaboration, we present promising preliminary concepts for streamlining the translation of research outcomes into everyday healthcare experiences. , -. CCS CONCEPTS• Human-centered computing → Empirical studies in HCI ; Accessibility technologies; • General and reference → Design.

Background: The evidence-based quantification of the relation between changes in movement quality and functionality can assist clinicians in achieving more effective structuring or adaptations of therapy. Facilitating this quantification through computational tools can also result in the generation of large-scale data sets that can inform automated assessment of rehabilitation. Interpretable automated assessment can leave more time for clinicians to focus on treatment and allow for remotely supervised therapy at the home. Methods: In our first experiment, we developed a rating process and accompanying computational tool to assist clinicians in following a standardized movement assessment process relating functionality to movement quality. We conducted three studies with three different versions of the computational rating tool. Clinicians rated task, segment, and movement feature performance for 440 videos in which stroke survivors executed standardized upper extremity therapy tasks related to functional activities. In our second experiment, we used the 440 rated videos, in addition to 140 videos of unimpaired subjects performing the same tasks, to improve our previously developed automated assessment ensemble model that automatically generates segmentation times and task ratings across the impaired and unimpaired movement. The automated assessment ensemble integrates expert knowledge constraints into data-driven training through a combination of HMM, transformer, MSTCN++, and decision tree computational modules. In our third experiment, we used the therapist and automated ratings to develop a four-layer Hierarchical Bayesian Model (HBM) for computing the statistical relation of movement quality changes to functionality. We first calculated conditional layer probabilities using clinician ratings of task, segment, and movement features. We increased the granularity of observation of the HBM by formulating ΔHBM, a correlation graph between kinematics and movement composite features. Finally, we used k-means clustering on the ΔHBM to identify three clusters of features among the 16 movement composite and 20 kinematic features and used the centroid of these clusters as the weights of the input data to our computational assessment ensemble. Results: We evaluated the efficacy of our rating interface in terms of inter-rater reliability (IRR) across tasks, segments, and movement features. The third version of the interface produced an average IRR of 67%, while the time per session (TPS) was the lowest of the three studies. By analyzing the ratings, we were able to identify a small number of movement features that have the highest probability of predicting functional improvement. We evaluated the performance of our automated assessment model using 60% impaired and 40% unimpaired movement data and achieved a frame-wise segmentation accuracy of 87.85±0.58 and a block-segmentation accuracy of 98.46±1.6. We also demonstrated the performance of our proposed HBM in correlation to clinician ratings with a correlation of over 90%. The HBM also generates a correlation graph, ΔHBM that relates 16 composite movement features to the 20 kinematic features. We can thus integrate the HBM into the computational assessment ensemble to perform automated and integrated movement quality and functionality assessment that is driven by computationally extracted kinematics. Conclusions: Combining standardized clinician ratings of videos with knowledge based and data-driven computational analysis of rehabilitation movement allows the expression of an HBM that increases the observability of the relation of movement quality to functionality and enables the training of computational algorithms for automated assessment of rehabilitation movement. While our work primarily focuses on the upper extremity of stroke survivors, the models can be adopted to many other neurorehabilitation contexts.

We are developing a system for long-term Semi-Automated Rehabilitation At the Home (SARAH) that relies on low-cost and unobtrusive video-based sensing. We present a cyber-human methodology used by the SARAH system for automated assessment of upper extremity stroke rehabilitation at the home. We propose a hierarchical model for automatically segmenting stroke survivor's movements and generating training task performance assessment scores during rehabilitation. The hierarchical model fuses expert therapist knowledge-based approaches with data-driven techniques. The expert knowledge is more observable in the higher layers of the hierarchy (task and segment) and therefore more accessible to algorithms incorporating high-level constraints relating to activity structure (i.e. type and order of segments per task). We utilize an HMM and a Decision Tree model to connect these high-level priors to data-driven analysis. The lower layers (RGB images and raw kinematics) need to be addressed primarily through data-driven techniques. We use a transformer-based architecture operating on low-level action features (tracking of individual body joints and objects) and a Multi-Stage Temporal Convolutional Network(MS-TCN) operating on raw RGB images. We develop a sequence combining these complementary algorithms effectively, thus encoding the information from different layers of the movement hierarchy. Through this combination, we produce robust segmentation and task assessment results on noisy, variable, and limited data, which is characteristic of low-cost video capture of rehabilitation at the home. Our proposed approach achieves 85% accuracy in per-frame labeling, 99% accuracy in segment classification, and 93% accuracy in task completion assessment. Although the methodology proposed in this paper applies to upper extremity rehabilitation using the SARAH system, it can potentially be used, with minor alterations, to assist automation in many other movement rehabilitation contexts (i.e. lower extremity training for neurological accidents).