3DPalsyNet: A Facial Palsy Grading and Motion Recognition Framework Using Fully 3D Convolutional Neural Networks

Abstract

The capability to perform facial analysis from video sequences has significant potential to positively impact in many areas of life. One such area relates to the medical domain to specifically aid in the diagnosis and rehabilitation of patients with facial palsy. With this application in mind, this paper presents an end-to-end framework, named 3DPalsyNet, for the tasks of mouth motion recognition and facial palsy grading. 3DPalsyNet utilizes a 3D CNN architecture with a ResNet backbone for the prediction of these dynamic tasks. Leveraging transfer learning from a 3D CNNs pre-trained on the Kinetics data set for general action recognition, the model is modified to apply joint supervised learning using center and softmax loss concepts. 3DPalsyNet is evaluated on a test set consisting of individuals with varying ranges of facial palsy and mouth motions and the results have shown an attractive level of classification accuracy in these tasks of 82% and 86% respectively. The frame duration and the loss function affect was studied in terms of the predictive qualities of the proposed 3DPalsyNet, where it was found shorter frame duration's of 8 performed best for this specific task. Centre loss and softmax have shown improvements in spatio-temporal feature learning than softmax loss alone, this is in agreement with earlier work involving the spatial domain.

Region Based Parallel Hierarchy Convolutional Neural Network for Automatic Facial Nerve Paralysis Evaluation

“…We incorporate LSTM in the network architecture. LSTM has been widely used in the task of facial expression and action recognition [29][30][31][32]. Zhang et al [31] combined the time and texture information of image sequences by Total paralysis Ⅲ…”

“…At present, there are few studies using image sequences for facial paralysis grading. Storey et al [32] proposed a 3D CNN architecture with a ResNet framework to recognize mouth motion and assess facial palsy. Different with facial expression recognition, FNP assessment is to evaluate the intensity of facial deformation through the landmark distance features and spatio-temporal variation information involved in patient's facial salient areas.…”

Deep Learning-Based Automated Forest Health Diagnosis From Aerial Images

“…The limits can be ascribed to two aspects. First, deep neural networks has been reported not robust to noises and other similar perturbations (adversary attacks) [45], while DNNs are somehow lack of explicit explanability unlike typical statistic approaches that model the data space explicitly [46] [53]. In our experiments, we have a limited amount of data samples.…”

Biometric Blockchain: A Secure Solution for Intelligent Vehicle Data Sharing

“…Classically, feature (such as LBP or SIFT) based [25,26] biometric verification is popular though it could be a bit timeconsuming. Recently new approaches such as deep neural networks [27][28][29][30] are taking over the field of biometric verification. Such biometric verification can be carried out on cloud servers, leading to a new topic called Biometrics-as-a-Service (BaaS).…”