Eyemotion: Classifying Facial Expressions in VR Using Eye-Tracking Cameras

Hickson, Steven; Dufour, Nick; Sud, Avneesh; Kwatra, Vivek; Essa, Irfan

doi:10.1109/wacv.2019.00178

Cited by 105 publications

(75 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Furthermore, deep learning techniques have been thoroughly applied by the participants of these two challenges (e.g., [240], [241], [242]). Additional related real-world applications, such as the Real-time FER App for smartphones [243], [244], Eyemotion (FER using eye-tracking cameras) [245], privacy-preserving mobile analytics [246], Unfelt emotions [247] and Depression recognition [248], have also been developed.…”

Section: Other Special Issuesmentioning

confidence: 99%

Deep Facial Expression Recognition: A Survey

Deng

2022

IEEE Trans. Affective Comput.

1,082

593

View full text Add to dashboard Cite

With the transition of facial expression recognition (FER) from laboratory-controlled to challenging in-the-wild conditions and the recent success of deep learning techniques in various fields, deep neural networks have increasingly been leveraged to learn discriminative representations for automatic FER. Recent deep FER systems generally focus on two important issues: overfitting caused by a lack of sufficient training data and expression-unrelated variations, such as illumination, head pose and identity bias. In this paper, we provide a comprehensive survey on deep FER, including datasets and algorithms that provide insights into these intrinsic problems. First, we introduce the available datasets that are widely used in the literature and provide accepted data selection and evaluation principles for these datasets. We then describe the standard pipeline of a deep FER system with the related background knowledge and suggestions of applicable implementations for each stage. For the state of the art in deep FER, we review existing novel deep neural networks and related training strategies that are designed for FER based on both static images and dynamic image sequences, and discuss their advantages and limitations. Competitive performances on widely used benchmarks are also summarized in this section. We then extend our survey to additional related issues and application scenarios. Finally, we review the remaining challenges and corresponding opportunities in this field as well as future directions for the design of robust deep FER systems.

show abstract

Section: Other Special Issuesmentioning

confidence: 99%

Deep Facial Expression Recognition: A Survey

Deng

2022

IEEE Trans. Affective Comput.

1,082

593

View full text Add to dashboard Cite

show abstract

“…The task is particularly challenging because we only have the user's eye regions as input rather than the entire face, as used in most emotive facial expressions classification benchmarks [26]. To our knowledge, only two previous studies have identified facial expressions from eyes, however, in these studies a view of the eyebrows was available [25] [16]. Eyebrows are considered important for emotive expression.…”

Section: Facial Expression Classificationmentioning

confidence: 99%

“…For the facial expression classification, we collected data from 15 subjects. We trained subjects to perform facial action units and labelled these data using the Facial Action Coding system, a widely recognized method for coding individual facial muscle movements [7]. These facial action units can then be mapped onto emotive expressions.…”

Section: Labelingmentioning

confidence: 99%

EyeNet: A Multi-Task Deep Network for Off-Axis Eye Gaze Estimation

Rajendran

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

Eye gaze estimation and simultaneous semantic understanding of a user through eye images is a crucial component in Virtual and Mixed Reality; enabling energy efficient rendering, multi-focal displays and effective interaction with 3D content. In head-mounted VR/MR devices the eyes are imaged off-axis to avoid blocking the user's gaze, this view-point makes drawing eye related inferences very challenging. In this work, we present EyeNet, the first single deep neural network which solves multiple heterogeneous tasks related to eye gaze estimation and semantic user understanding for an off-axis camera setting. The tasks include eye segmentation, blink detection, emotive expression classification, IR LED glints detection, pupil and cornea center estimation. To train EyeNet end-to-end we employ both hand labelled supervision and model based supervision. We benchmark all tasks on MagicEyes, a large and new dataset of 587 subjects with varying morphology, gender, skin-color, make-up and imaging conditions.

show abstract

“…They also depend on the features of the object of interest [4,5]. Accordingly, bottom-up and top-down approaches are mainly driven by the visual characteristics of a scene and the task of interest, respectively [6,7].…”

Section: Introductionmentioning

confidence: 99%

Visual Saliency Prediction Based on Deep Learning

2019

View full text Add to dashboard Cite

Human eye movement is one of the most important functions for understanding our surroundings. When a human eye processes a scene, it quickly focuses on dominant parts of the scene, commonly known as a visual saliency detection or visual attention prediction. Recently, neural networks have been used to predict visual saliency. This paper proposes a deep learning encoder-decoder architecture, based on a transfer learning technique, to predict visual saliency. In the proposed model, visual features are extracted through convolutional layers from raw images to predict visual saliency. In addition, the proposed model uses the VGG-16 network for semantic segmentation, which uses a pixel classification layer to predict the categorical label for every pixel in an input image. The proposed model is applied to several datasets, including TORONTO, MIT300, MIT1003, and DUT-OMRON, to illustrate its efficiency. The results of the proposed model are quantitatively and qualitatively compared to classic and state-of-the-art deep learning models. Using the proposed deep learning model, a global accuracy of up to 96.22% is achieved for the prediction of visual saliency.

show abstract

Eyemotion: Classifying Facial Expressions in VR Using Eye-Tracking Cameras

Cited by 105 publications

References 43 publications

Deep Facial Expression Recognition: A Survey

Deep Facial Expression Recognition: A Survey

EyeNet: A Multi-Task Deep Network for Off-Axis Eye Gaze Estimation

Visual Saliency Prediction Based on Deep Learning

Contact Info

Product

Resources

About