Detection of eye contact with deep neural networks is as accurate as human experts

Abstract

Eye contact is among the most primary means of social communication used by humans. Quantification of eye contact is valuable as a part of the analysis of social roles and communication skills, and for clinical screening. Estimating a subject’s looking direction is a challenging task, but eye contact can be effectively captured by a wearable point-of-view camera which provides a unique viewpoint. While moments of eye contact from this viewpoint can be hand-coded, such a process tends to be laborious and subjective. In this work, we develop a deep neural network model to automatically detect eye contact in egocentric video. It is the first to achieve accuracy equivalent to that of human experts. We train a deep convolutional network using a dataset of 4,339,879 annotated images, consisting of 103 subjects with diverse demographic backgrounds. 57 subjects have a diagnosis of Autism Spectrum Disorder. The network achieves overall precision of 0.936 and recall of 0.943 on 18 validation subjects, and its performance is on par with 10 trained human coders with a mean precision 0.918 and recall 0.946. Our method will be instrumental in gaze behavior analysis by serving as a scalable, objective, and accessible tool for clinicians and researchers.

Toward an Attentive Robotic Architecture: Learning-Based Mutual Gaze Estimation in Human–Robot Interaction

“…In this section, the mutual gaze classifier is compared with the solution proposed in Chong et al (2020) . To the best of our knowledge, this is the most recent solution in the current literature that best adapts to our purposes.…”

“…To the best of our knowledge, this is the most recent solution in the current literature that best adapts to our purposes. In Chong et al (2020) , the authors trained a deep convolution neural network (i.e., ResNet-50 ( He et al, 2016 )) as the backbone to automatically detect eye contact during face-to-face interactions. As network performance, the authors reported an overall precision of 0.94 and an F1-score of 0.94 on 18 validation subjects.…”

“…For example, in the context of human–human interaction, Chong et al (2020) proposed a novel approach based on deep neural networks to detect eye contact using PoV cameras with reliability equivalent to expert human raters. The proposed algorithm has been used in this work as the baseline for the comparison (see Section 4.3 ).…”

“…Such a module is then embedded into the iCub’s framework and validated both in silico and in online scenarios. Furthermore, we compared our method with the solution proposed in Chong et al (2020) , achieving an improvement in the accuracy of around 15 percentage points. 3.…”

Technical Challenges for Smooth Interaction With Seniors With Dementia: Lessons From Humanitude™

“…For a social robot, first-person videos can be captured by a camera embedded in the eye-pupil or the forehead. From these videos, face-to-face postures (distances or angles) or eye contact states can be obtained using facial detection and/or machine learning techniques ( Chong et al, 2020 ; Mitsuzumi et al, 2017 ). Similar systems such as wearable eye trackers ( Chong et al, 2017 ) or proximity sensors ( Hachisu et al, 2018 ) can be used for face-to-face interaction analysis as well.…”