Deep Hierarchical Network With Line Segment Learning for Quantitative Analysis of Facial Palsy

Abstract

We propose the deep hierarchical network (DHN) for the quantitative analysis of facial palsy. Facial palsy, also known as Bell's palsy, is the most common type of facial nerve palsy that results in the loss of muscle control in the affected facial regions. Typical symptoms include facial deformity and facial expression dysfunction. To the best of our best knowledge, all approaches for the automatic detection of facial palsy consider hand-crafted features. This paper reports the first deep-learning-based approach developed for the real-time quantitative analysis of facial palsy. The proposed DHN consists of three component networks: the first detects the subject's face, the second detects the facial landmarks and line segments on the detected face, and the third detects the local palsy regions. The first component network is built on the YOLO2 detector. The second component network is developed on a fused network architecture that incorporates a line segment learning network for locating the facial landmarks and line segments. The third component network is developed on an object detection network with the line-segment-embedded input that combines the landmarked region and the line segments detected by the second component network. The novelties of this research include: 1) the modification of a state-of-the-art edge detector for extracting the facial line segments; 2) the embedding of the line segment learning for the detection of facial landmarks and local palsy regions; 3) the quantitative description of the facial palsy syndrome intensity; and 4) the release of the first clinically labeled database, the YouTube Facial Palsy (YFP) database. The making of the YFP database solves the issue that previous methods were all evaluated on proprietary databases, making the comparison of different methods extremely difficult. The YFP database includes 32 videos of 21 patients collected from YouTube and labeled by clinic specialists. To enhance the robustness against facial expression variations, we include the CK+ facial expression database in the training. We show that the proposed DHN not only just detects the local palsy regions but also captures the intensity of the facial palsy syndrome over time, enabling the quantitative description of the syndrome. The experiments show that the proposed approach offers an accurate and efficient real-time solution for facial palsy analysis.INDEX TERMS Facial palsy, bell palsy, medical image diagnosis, face alignment, face recognition.

Region Based Parallel Hierarchy Convolutional Neural Network for Automatic Facial Nerve Paralysis Evaluation

“…All training FNP image sequences and normal face frames are from the public available YFP Database [20], [21], and Extended CohnKanade (CK+) Database [22]. The quantitative outcome in [21] is just a regional characteristic, not a comprehensive diagnosis. The detection accuracy of landmark location was directly related to the recognition accuracy of intensity outcomes.…”

“…Song et al [27] proposed a neural network model called Inception-DeepID-FNP for classifying seven facial movements. Hsu et al [21] quantitatively analyzed the intensity variation over time by using facial image sequences and the landmark outline. The accuracy of landmark directly affects the line segment map.…”

“…Some samples of different HBS grades and different facial movements are shown in Fig. 1, including closing eyes, frowning, wrinkling nose, lifting corners of mouth and pursing lips [20][21][22]. These movements illustrate paralysis feature in different facial areas.…”

A Case Study of Facial Emotion Classification Using Affdex

“…The facial polio causes loss of muscle control in the affected areas, which represents not only face deformity but especially facial expression dysfunction and thus the inability of most current algorithms to capture the patient’s true emotions. According to the findings of Hsu et al [70], current approaches to the automatic detection of polio in childhood take into account in the most cases manual functions, resulting in incorrect classification of patients’ emotional status.…”

Automatic Facial Paralysis Evaluation Augmented by a Cascaded Encoder Network Structure

“…Therefore, most existing facial paralysis evaluation methods target at identifying the spatial information like asymmetries demonstrated in the paralysed faces. For example, Hsu et al [17] presented an deep network for facial palsy analysis. Their method relied on the facial landmark localization based on line segmentation strategy.…”