Hand2Face: Automatic synthesis and recognition of hand over face occlusions

Abstract

A person's face discloses important information about their affective state. Although there has been extensive research on recognition of facial expressions, the performance of existing approaches is challenged by facial occlusions. Facial occlusions are often treated as noise and discarded in recognition of affective states. However, hand over face occlusions can provide additional information for recognition of some affective states such as curiosity, frustration and boredom. One of the reasons that this problem has not gained attention is the lack of naturalistic occluded faces that contain hand over face occlusions as well as other types of occlusions. Traditional approaches for obtaining affective data are time demanding and expensive, which limits researchers in affective computing to work on small datasets. This limitation affects the generalizability of models and deprives researchers from taking advantage of recent advances in deep learning that have shown great success in many fields but require large volumes of data. In this paper, we first introduce a novel framework for synthesizing naturalistic facial occlusions from an initial dataset of non-occluded faces and separate images of hands, reducing the costly process of data collection and annotation. We then propose a model for facial occlusion type recognition to differentiate between hand over face occlusions and other types of occlusions such as scarves, hair, glasses and objects. Finally, we present a model to localize hand over face occlusions and identify the occluded regions of the face.

Prediction of the VDT Worker's Headache Using Convolutional Neural Network with Class Activation Mapping

“…Therefore, there is no medical standard database including a human action as a sign of headache. In this work, we compared the proposed method with three methods using a face detection, HOG, and SVM [8], Force field method and Camshift [10], and CNN with divided the face into 8 regions [11] as shown in Table II. In the previous methods, face detection and head orientation directly affect to the post‐processing [8–10].…”

“…In the previous methods, face detection and head orientation directly affect to the post‐processing [8–10]. The detection of the border of the face and nose is difficult even though the deep learning applied in Hand2face method [11]. Table III shows a statistical comparison between the proposed method and the Hand2face method.…”

“…presented an improved method divided a face into eight regions. It applied a convolutional neural network for each small region [11]. This method also has a limitation at detecting the hand over face poses, especially failed in the face borders and nose area.…”

Fake It Till You Make It: Face analysis in the wild using synthetic data alone

“…Zhu et al [79] fit a 3DMM to face images, and warped them to augment the head pose. Nojavanasghari et al [42] composited hand images onto faces to improve face detection. These approaches can only make minor adjustments to existing images, limiting their use.…”

Extended Labeled Faces in-the-Wild (ELFW): Augmenting Classes for Face Segmentation

“…Blending source hands into faces requires realistic color and pose matching with respect to the targeted face. To the extent of our knowledge, the most determined work on this regard is Hand2Face [52], whose authors gave especial relevance to hands because their pose discloses relevant information about the person's affective state. Other hand datasets made of images captured in first person view, such as Egohands [2], GTEA [38], or EgoYouTubeHands [72] are not suitable to be attached to faces in a natural way.…”