Design and application of facial expression analysis system in empathy ability of children with autism spectrum disorder

Abstract

Empathy is an important social ability in the early childhood development. One of the significant characteristics of children with autism spectrum disorder (ASD) is their lack of empathy, which makes it difficult for them to feel and understand other people's emotions and to judge other people's behavioral intentions, leading to social disorders. This research designs and implements a facial expression analysis system that could obtain and analyze the real-time facial expressions of children when viewing stimulus materials, and then evaluate the differences of empathy ability between ASD children and typical development (TD) children. The results of this research provide new ideas for the evaluation of ASD children, and also help to develop empathy intervention plans for ASD children.

Discriminative Few Shot Learning of Facial Dynamics in Interview Videos for Autism Trait Classification

“…The failure to accurately interpret facial expressions (that is, happiness, surprise, fear, anger, disgust, sadness) [67], [68], [69], [70] and facial processing [71], [72], [73] is one of the key impairments in ASD. Recent work also indicates that observers with ASD have difficulty using facial motion patterns to judge identity or [26] 2020 adult gaze pattern image 15ASD+15TD Scanpath Trend Analysis Ahuja et al [27] 2020 adult gaze pattern video 35ASD+25TD Gaze features Tao&Shyu [22] 2019 child gaze pattern image 14ASD+14TD CNN+LSTM Duan et al [21] 2018 child gaze pattern image 13ASD GAN Dris et al [19] 2019 child gaze pattern image -Region of Interests Wei et al [20] 2019 child gaze pattern image -CNN Li et al [28] 2018 child gaze pattern video 53ASD+136TD Displacement Feature Wan et al [29] 2019 child gaze pattern image 37ASD+37TD Areas of Interest Fernández et al [30] 2020 child gaze pattern image 8ASD+23TD CNN Liu et al [31] 2016 child gaze pattern on face image 29ASD+2groups TD K-means Jiang et al [32] 2019 -gaze pattern on face image 23ASD+35TD DNN Kaliukhovich et al [33] 2021 child+adult gaze pattern images 94ASD+38TD -Leo et al [34] 2019 child facial expression image 17ASD+10TD CNN Beary et al [35] 2020 child facial expression image 1,507ASD+1,507TD MobileNet Akter et al [36] 2021 child facial expression image 1,468ASD+1,468TD MobileNet-V1 Lu& Perkowski [37] 2021 child facial features image 561ASD+561TD VGG16 Kowallik et al [38] 2021 adult facial expression image 55ASD logistic regression Lecciso et al [39] 2021 child facial expression image 12ASD -Guo et al [40] 2021 child facial expression image 30ASD+30TD -Elshoky et al [41] 2021 child facial features image 2,936 A set of ML methods Bangerter et al [42] 2020 child+adult facial expression video 124ASD+41NT Gaussian Mixture Model Banire et al [43] 2021 child facial expression video 20ASD+26TD CNN Zlibut et al [44] 2021 adult f...…”

“…In [35], a MobileNet-based deep learning model was introduced to classify children with ASD. [40] proposed a facial expression analysis system to evaluate differences in empathy ability between children with ASD and TD by analyzing real-time facial expressions of children. [37] designed a VGG16 transfer learning-based ASD screening solution to detect ASD using facial images on a unique ASD dataset of clinically diagnosed children.…”

Automatic Emotion Recognition in Children with Autism: A Systematic Literature Review

“…The emotions were grouped regarding their similarity. [21,24] attention (1): [25] bored (1): [25] fear-related 16 fear ( 12): [17,[26][27][28][29][30][31][32][33][34][35][36] anxiety (3): [20,22,25] trepidation (1): [37] joy-related 21 joy ( 5): [21,[26][27][28][29] liking (3): [19,20,22] happiness (12): [17,24,25,[30][31][32][33][34][35][37][38][39] smile (1): [40] relaxation (1): [37] There are studies that treat attention, engagement, involvement, and boredom as emotional states. They are more like mental states; however, they are emotion-related and were used in the context of emotion recognition in the analyzed papers.…”

“…joy 21 [17,[19][20][21][22][24][25][26][27][28][29][30][31][32][33][34][35][36][37][38][39] anger 12 [17,[27][28][29][30][31][32][33]35,36,38,39] fear 12 [17,[26][27][28][29][30][31][32][33][34][35][36] disgust 7 [17,25,27,29,32,35,36] sadness 16 [17,…”

“…Number of Papers Papers joy 3 [19,20,22] joy, neutral 2 [21,24] joy, sadness 2 [21,37] joy, fear, sadness 2 [26,34] joy, disgust, sadness, neutral 1 [25] joy, anger, sadness, neutral 2 [38,39] joy, anger, fear, sadness, neutral 2 [28,30] joy, anger, fear, disgust, sadness, surprise 1 [17] joy, anger, fear, sadness, surprise, neutral 2 [31,33] joy, anger, fear, disgust, sadness, neutral 1 [36] joy, anger, fear, disgust, sadness, surprise, netural 4 [27,29,32,35] In some papers, only two-class recognition was performed. In those papers, joy is recognized as a class combined with no-joy in three papers, with a neutral state in two papers, and with sadness in two papers.…”