Automated Computer Vision Assessment of Hypomimia in Parkinson Disease: Proof-of-Principle Pilot Study

Abstract

Background Facial expressions require the complex coordination of 43 different facial muscles. Parkinson disease (PD) affects facial musculature leading to “hypomimia” or “masked facies.” Objective We aimed to determine whether modern computer vision techniques can be applied to detect masked facies and quantify drug states in PD. Methods We trained a convolutional neural network on images extracted from videos of 107 self-identified people with PD, along with 1595 videos of controls, in order to detect PD hypomimia cues. This trained model was applied to clinical interviews of 35 PD patients in their on and off drug motor states, and seven journalist interviews of the actor Alan Alda obtained before and after he was diagnosed with PD. Results The algorithm achieved a test set area under the receiver operating characteristic curve of 0.71 on 54 subjects to detect PD hypomimia, compared to a value of 0.75 for trained neurologists using the United Parkinson Disease Rating Scale-III Facial Expression score. Additionally, the model accuracy to classify the on and off drug states in the clinical samples was 63% (22/35), in contrast to an accuracy of 46% (16/35) when using clinical rater scores. Finally, each of Alan Alda’s seven interviews were successfully classified as occurring before (versus after) his diagnosis, with 100% accuracy (7/7). Conclusions This proof-of-principle pilot study demonstrated that computer vision holds promise as a valuable tool for PD hypomimia and for monitoring a patient’s motor state in an objective and noninvasive way, particularly given the increasing importance of telemedicine.

Automated video-based assessment of facial bradykinesia in de-novo Parkinson’s disease

“…The strength of this study is that we focused on assessing hypomimia via freely spoken monologue, which represents the most natural and available source of facial movements. The assessment of freely spoken monologue holds the greatest potential as the most convenient hypomimia measurement, which could be easily implemented in video communication systems and therefore scalable to a large population 16 . In addition, the freely spoken monologue extends the assessment of deliberate facial movement by the involvement of spontaneous facial expressions as a constituent of nonverbal communication and therefore provides a comprehensive picture of the hypomimia impact.…”

“…Therefore, the detailed analysis of different movement types is a matter of future research based on specialized mimic tasks. Yet, pilot studies aimed at an objective assessment of deliberate facial movement support the effect of PD on reduced speed and peak amplitude of the intentional facial grimacing 16 – 29 . On the other hand, the static assessment of facial shape provided only limited results supporting dynamic objective assessment as a superior approach 30 .…”

Automated Quantification of Eye Tics Using Computer Vision and Deep Learning Techniques

“…Fourth, the algorithm is tested on new unseen datasets to examine validity (ie, correspondence between human coded "ground truth" and algorithm). This approach has been used to automate video-based detection of human motion [27][28][29][30][31] and behaviors relevant to autism, 32,33 obsessive-compulsive disorder, 34 neurodevelopmental risk, 35 Parkinson's disease, 36 bradykinesia, 37,38 hypomimia, 39 seizure, 40 infant neuromotor impairment, 41 and depression. 42 Fully automated measurement has several advantages: algorithms are efficient, immune to human limitations (eg, training burden, drift, reactivity, fatigue, distraction), and can measure aspects of movement that humans struggle to quantify (eg, movement amplitude and velocity).…”

“…Fourth, the algorithm is tested on new unseen datasets to examine validity (ie, correspondence between human coded “ground truth” and algorithm). This approach has been used to automate video‐based detection of human motion 27‐31 and behaviors relevant to autism, 32,33 obsessive‐compulsive disorder, 34 neurodevelopmental risk, 35 Parkinson's disease, 36 bradykinesia, 37,38 hypomimia, 39 seizure, 40 infant neuromotor impairment, 41 and depression 42 …”

Acoustic and Facial Features From Clinical Interviews for Machine Learning–Based Psychiatric Diagnosis: Algorithm Development

“…In recent years, progress has been made in audiovisual data processing [13][14][15][16][17][18][19][20][21]. Advances in this technology could play a pivotal role in supporting automated methods of collecting objective adjunctive patient data to inform diagnostic procedures, psychiatric symptom identification, and psychiatric symptom monitoring.…”