Facial expression recognition for monitoring neurological disorders based on convolutional neural network

Yolcu, Gozde; Öztel, İsmail; Kazan, Serap; Öz, Cemil; Palaniappan, Kannappan; Lever, Teresa E.; Bunyak, Filiz

doi:10.1007/s11042-019-07959-6

Cited by 79 publications

(31 citation statements)

References 69 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The alternating structure of multiple convolutional and nonlinear activation layers makes extracting deeper and better features than a single convolutional layer structure. In the ILSVRC2015 competition, ResNet, proposed by scholars, won the championship [5]. A connection method called short cut connection in ResNet can theoretically keep the network in an optimal state while the network layer is constantly deepening.…”

Section: Introductionmentioning

confidence: 99%

[Retracted] Facial Expression Recognition Based on Convolutional Neural Network Fusion SIFT Features of Mobile Virtual Reality

Yao

Qiu

2021

Wireless Communications and Mobile Computing

View full text Add to dashboard Cite

Facial expression recognition computer technology can obtain the emotional information of the person through the expression of the person to judge the state and intention of the person. The article proposes a hybrid model that combines a convolutional neural network (CNN) and dense SIFT features. This model is used for facial expression recognition. First, the article builds a CNN model and learns the local features of the eyes, eyebrows, and mouth. Then, the article features are sent to the support vector machine (SVM) multiclassifier to obtain the posterior probabilities of various features. Finally, the output result of the model is decided and fused to obtain the final recognition result. The experimental results show that the improved convolutional neural network structure ER2013 and CK+ data sets’ facial expression recognition rate increases by 0.06% and 2.25%, respectively.

show abstract

Section: Introductionmentioning

confidence: 99%

[Retracted] Facial Expression Recognition Based on Convolutional Neural Network Fusion SIFT Features of Mobile Virtual Reality

Yao

Qiu

2021

Wireless Communications and Mobile Computing

View full text Add to dashboard Cite

show abstract

“…The potential of artificial intelligence (AI)-based facial expression analysis using a facial expression recognition system (FERS) to identify emotions, pain, and nonverbal information among persons with psychiatric disorders has been documented [6][7][8][9]. FERS successfully predicted 8 basic mood phenotypes using more than 1,000,000 facial images collected from the internet, i.e., disgust, fear, sadness, anger, happiness, surprise, neutral, and contempt [9][10][11]. The accuracy of FERS based on a convolutional neural network (CNN) to recognize these 8 emotional expressions was approximately 87.7-94.2%, which was noninferior to that of the support vector machine (SVM) method (77.1-92.8%) [9][10][11].…”

Section: Introductionmentioning

confidence: 99%

“…FERS successfully predicted 8 basic mood phenotypes using more than 1,000,000 facial images collected from the internet, i.e., disgust, fear, sadness, anger, happiness, surprise, neutral, and contempt [9][10][11]. The accuracy of FERS based on a convolutional neural network (CNN) to recognize these 8 emotional expressions was approximately 87.7-94.2%, which was noninferior to that of the support vector machine (SVM) method (77.1-92.8%) [9][10][11]. Although deficient facial expressions were common presentations of persons with neurodegenerative disorder, the enhanced facial responses to pain in PwD provided opportunities for FERS to identify somatic discomforts [8,10,12].…”

Section: Introductionmentioning

confidence: 99%

Predicting neuropsychiatric symptoms of persons with dementia in a day care center using a facial expression recognition system

Chen

Tsai

et al. 2022

Aging

View full text Add to dashboard Cite

Background: Behavioral and psychological symptoms of dementia (BPSD) affect 90% of persons with dementia (PwD), resulting in various adverse outcomes and aggravating care burdens among their caretakers. This study aimed to explore the potential of artificial intelligence-based facial expression recognition systems (FERS) in predicting BPSDs among PwD. Methods: A hybrid of human labeling and a preconstructed deep learning model was used to differentiate basic facial expressions of individuals to predict the results of Neuropsychiatric Inventory (NPI) assessments by stepwise linear regression (LR), random forest (RF) with importance ranking, and ensemble method (EM) of equal importance, while the accuracy was determined by mean absolute error (MAE) and root-mean-square error (RMSE) methods. Results: Twenty-three PwD from an adult day care center were enrolled with ≥ 11,500 FERS data series and 38 comparative NPI scores. The overall accuracy was 86% on facial expression recognition. Negative facial expressions and variance in emotional switches were important features of BPSDs. A strong positive correlation was identified in each model (EM: r = 0.834, LR: r = 0.821, RF: r = 0.798 by the patientwise method; EM: r = 0.891, LR: r = 0.870, RF: r = 0.886 by the MinimPy method), and EM exhibited the lowest MAE and RMSE. Conclusions: FERS successfully predicted the BPSD of PwD by negative emotions and the variance in emotional switches. This finding enables early detection and management of BPSDs, thus improving the quality of dementia care.

show abstract

“…A method capable of objectively characterizing variations in naturally occurring facial expressions that vary with disease progression would allow the patient state to be continuously evaluated outside of a clinical setting, opening up the possibility of remote or telemedicine-based monitoring. Video analysis has started to demonstrate success in objectively quantifying emotions in psychiatry [15,16] and neurology [17].…”

Section: Introductionmentioning

confidence: 99%

Automated Computer Vision Assessment of Hypomimia in Parkinson Disease: Proof-of-Principle Pilot Study

Abrami¹,

Gunzler²,

Kilbane³

et al. 2021

J Med Internet Res

View full text Add to dashboard Cite

Background Facial expressions require the complex coordination of 43 different facial muscles. Parkinson disease (PD) affects facial musculature leading to “hypomimia” or “masked facies.” Objective We aimed to determine whether modern computer vision techniques can be applied to detect masked facies and quantify drug states in PD. Methods We trained a convolutional neural network on images extracted from videos of 107 self-identified people with PD, along with 1595 videos of controls, in order to detect PD hypomimia cues. This trained model was applied to clinical interviews of 35 PD patients in their on and off drug motor states, and seven journalist interviews of the actor Alan Alda obtained before and after he was diagnosed with PD. Results The algorithm achieved a test set area under the receiver operating characteristic curve of 0.71 on 54 subjects to detect PD hypomimia, compared to a value of 0.75 for trained neurologists using the United Parkinson Disease Rating Scale-III Facial Expression score. Additionally, the model accuracy to classify the on and off drug states in the clinical samples was 63% (22/35), in contrast to an accuracy of 46% (16/35) when using clinical rater scores. Finally, each of Alan Alda’s seven interviews were successfully classified as occurring before (versus after) his diagnosis, with 100% accuracy (7/7). Conclusions This proof-of-principle pilot study demonstrated that computer vision holds promise as a valuable tool for PD hypomimia and for monitoring a patient’s motor state in an objective and noninvasive way, particularly given the increasing importance of telemedicine.

show abstract

Facial expression recognition for monitoring neurological disorders based on convolutional neural network

Cited by 79 publications

References 69 publications

[Retracted] Facial Expression Recognition Based on Convolutional Neural Network Fusion SIFT Features of Mobile Virtual Reality

[Retracted] Facial Expression Recognition Based on Convolutional Neural Network Fusion SIFT Features of Mobile Virtual Reality

Predicting neuropsychiatric symptoms of persons with dementia in a day care center using a facial expression recognition system

Automated Computer Vision Assessment of Hypomimia in Parkinson Disease: Proof-of-Principle Pilot Study

Contact Info

Product

Resources

About