Video-Based Activity Recognition for Automated Motor Assessment of Parkinson's Disease

Abstract

Over the last decade, video-enabled mobile devices have become ubiquitous, while advances in markerless pose estimation allow an individual's body position to be tracked accurately and efficiently across the frames of a video. Previous work by this and other groups has shown that pose-extracted kinematic features can be used to reliably measure motor impairment in Parkinson's disease (PD). This presents the prospect of developing an asynchronous and scalable, video-based assessment of motor dysfunction. Crucial to this endeavour is the ability to automatically recognise the class of an action being performed, without which manual labelling is required.Representing the evolution of body joint locations as a spatio-temporal graph, we implement a deep-learning model for video and frame-level classification of activities performed according to part 3 of the Movement Disorder Society Unified PD Rating Scale (MDS-UPDRS). We train and validate this system using a dataset of n=7310 video clips, recorded at 5 independent sites. This approach reaches human-level performance in detecting and classifying periods of activity within monocular video clips. Our framework could support clinical workflows and patient care at scale through applications such as quality monitoring of clinical data collection, automated labelling of video streams, or a module within a remote selfassessment system.

Artificial intelligence-based video monitoring of movement disorders in the elderly: a review on current and future landscapes

“…Ten articles were finally selected for review [ Table 1 ]. [ 21 22 23 24 25 26 27 28 29 30 ] The selected studies report the accuracy and agreement with the clinician rating of the predicted UPDRS score, binary classification between PD and healthy subjects, or tremor detection. [ 29 ]…”

“…It is notable that out of the four key elements of bradykinesia, studies have started to include the other well-known clinical features of halts or hesitations of upper limb movement only recently. [ 21 22 ] This may be because so far, there has been no clear mathematical description or definition by MDS to describe the paroxysmal freezing phenomena of the upper limb. However, such recent trend reflects that researchers are now placing stronger emphasis on the comprehensiveness of their models to better align conceptually with UPDRS guidelines.…”

Clinically Informed Automated Assessment of Finger Tapping Videos in Parkinson’s Disease

“…This system is now being integrated into automatic PD classification research [27][28][29]. However, a large portion of research still relies on readily available consumer cameras, such as standard camcorders and smartphone cameras, for daily clinical practice [4,[30][31][32]. Regarding the categorization of these studies, they can be either marker-based Motion Capture systems (MoCaps) or markerless, determined by whether or not they use body-attached markers.…”

“…Recognizing this, recent research has increasingly prioritized the extraction of more-comprehensive features and the interpretability of the resulting models. Morinan et al [4,30] introduced a Random Forest algorithm for classifying bradykinesia, utilizing 11 kinetic features derived from video recordings of 1156 MDS-UPDRS assessments across five medical centers. Similarly, Islam et al [31] selected clinically pertinent features from an initial set of 53, explaining their model's performance by employing the Shapley Additive Explanations method, alongside various regression models.…”

Interpretable Video-Based Tracking and Quantification of Parkinsonism Clinical Motor States

“…Finally, these tools often focus on prolonged videos from a formal clinical examination or features from a single motor modality, increasing the burden of video acquisition and missing the opportunity to integrate different domains (e.g., body posture, hand movement, facial expression), which has the potential to significantly increase the accuracy and robustness of ML predictions [23][24][25][26] . They also typically only predict metrics directly corresponding to a single modality (i.e., predict only MDS-UPDRS finger tapping score) 22,27 . A truly valuable video-based solution for tracking PD motor symptom progression would need to be affordable, accessible, automated, transparent, and able to obtain rich and clinically relevant metrics for holistic evaluation of PD symptoms 28,29 .…”