Automatic Facial Paralysis Evaluation Augmented by a Cascaded Encoder Network Structure

Abstract

Facial paralysis refers to a facial nerve disordering, with which people may lose the abilities to accurately control their facial muscles for certain facial performances. The diagnosis of such disordering is mainly based on the observation of patient's face in terms of the facial spatial information, such as facial asymmetry. Up to now, this area is still dominated by therapists' subjective examinations clinically. Therefore, automations for this task receive wide attentions in both academic and industrial fields. Recently, the deep learning based methods, the convolutional neural networks (CNNs) more specifically, demonstrate their competitive performance compared with traditional approaches in many areas. However, due to the lack of the structured/labelled facial paralysis data as training data, those deep learning based solutions are still not able to fully attach their superiorities to the facial paralysis evaluation tasks. Another essential aspect for automation in facial paralysis analysis is the facial spatial information extraction. Semantic segmentation is a better choice than traditional template-based facial landmark detection for analysing facial paralysis images, which contain faces in uncommon patterns. However, most existing semantic segmentation approaches are made for indoor or outdoor scene parsing. To this end, this paper presents a deep learning-based approach for automatic facial paralysis grading prediction. The proposed model utilizes a cascaded encoder structure, which explores the advantages of the facial semantic feature for facial spatial information extraction, and then benefits the facial paralysis assessment. A dual-stage cascaded training process is adopted to utilize a mixture of normal and paralysed faces as training data, which exports a well-trained deep neural network model for facial paralysis evaluation. Experiments are conducted in two aspects to demonstrate the performance of each components of the proposed model. Encouraging results are illustrated compared with several existing approaches in the related areas.INDEX TERMS Facial nerve paralysis, dual-stage, cascaded neural network, facial attribute segmentation, paralysis grading prediction.

Region Based Parallel Hierarchy Convolutional Neural Network for Automatic Facial Nerve Paralysis Evaluation

“…Dynamic facial motion features can be extracted through multiple consecutive image frames, which are beneficial to recognizing facial muscle movements. Recent development of computer vision and deep learning networks have demonstrated revolutionary capabilities in feature learning and image recognition, especially in medical imaging [16]- [19]. Some samples of different HBS grades and different facial movements are shown in Fig.…”

The Feasibility of An Automatical Facial Evaluation System Providing Objective and Reliable Results for Facial Palsy

“…Besides, with the simple evaluation process, patients can complete AFES evaluation independently without relying on clinicians which suggested the possibility of applying AFES for remote medical evaluation and treatment. There is potential for AFES to be deployed onto portable devices like smartphones, which will expand its applications [65]. With the extra indicators offered by AFES, the patients can monitor the specific changes of facial motor function in real time and at any time, which might help patients establish a correct perception of their facial function and alleviate their psychological problems when they stay at home [66], [67].…”

Objective grading facial paralysis severity using a dynamic 3D stereo photogrammetry imaging system

“…Objective assessments are usually achieved by quantifying facial movements of facial images [13][14][15][16], videos [17,18], 3D facial expressions [19][20][21][22]. These imaging devices are more accessible than those obtrusive physical interventions devices such as electroneurography [23] and electromyography [24], or motion tracking of facial markers [25,26].…”

“…There was a gap between the quantifying the facial asymmetry and the clinical needs of diagnosis of the severity of the facial paralysis. A few studies [18,[29][30][31][32] applied machine learning techniques to quantify the severity based on facial asymmetry measurements from videos or static images. An Artificial Neural Network [ANN] took relevant extracted information from the video feed and quickly estimated a score of the current facial nerve damage [18].…”

“…Their database consists of 593 sequences generated from 113 subjects, while the facial palsy data set consists of 696 different sequences with 17 subjects collected from online sources. Wang et al [31] proposed model utilizing a cascaded encoder structure, which explored the advantages of the facial semantic feature for facial spatial information extraction, and then benefited the facial paralysis assessment. They had collected around 25000 facial images with the facial attribute markup ground truth, around 12000 facial paralysis images to form up a facial paralysis dataset, and manually marked up the facial features in the images.…”