Detecting Gaze Towards Eyes in Natural Social Interactions and Its Use in Child Assessment

Abstract

Eye contact is a crucial element of non-verbal communication that signi es interest, a ention, and participation in social interactions. As a result, measures of eye contact arise in a variety of applications such as the assessment of the social communication skills of children at risk for developmental disorders such as autism, or the analysis of turn-taking and social roles during group meetings. However, the automated measurement of visual a ention during naturalistic social interactions is challenging due to the di culty of estimating a subject's looking direction from video. is paper proposes a novel approach to eye contact detection during adult-child social interactions in which the adult wears a point-of-view camera which captures an egocentric view of the child's behavior. By analyzing the child's face regions and inferring their head pose we can accurately identify the onset and duration of the child's looks to their social partner's eyes. We introduce the Pose-Implicit CNN, a novel deep learning architecture that predicts eye contact while implicitly estimating the head pose. We present a fully automated system for eye contact detection that solves the sub-problems of end-to-end feature learning and pose estimation using deep neural networks. To train our models, we use a dataset comprising 22 hours of 156 play session videos from over 100 children, half of whom are diagnosed with Autism Spectrum Disorder. We report an overall precision of 0.76, recall of 0.80, and an area under the precision-recall curve of 0.79, all of which are signi cant improvements over existing methods.

Computer vision in autism spectrum disorder research: a systematic review of published studies from 2009 to 2019

“…Most of the studies have been conducted in a highly controlled environment in which the subjects were asked to view a screen for a short period of time. Recently, Chong et al 78 presented a novel deep learning architecture for eye contact detection in natural social interactions. In their study, eye contact detection was performed during adult-child sessions in which the adult wears a point-of-view camera.…”

Visual-Based Eye Contact Detection in Multi-Person Interactions

“…Eye contact detection is a binary decision on whether someone gaze falls onto target (e.g., face, screen) or not. Many methods have been developed to handle this issue by either using a head-mounted device [12], [13] or requiring LEDs attached to the target [14]. To avoid the intrusive devices, more works focus on developing methods that do not require any intrusive device such as the work of Smith et al [15] as they have used a classification approach to determine eye contact with a camera, but their method requires prior knowledge about the size and location of the target.…”

Analyzing the Use of Camera Glasses in the Wild

“…Although there is not much research specifically on Spectacles, it is worth noting that prior work on smart glasses referred to Spectacles as a tool for lifelogging [7,9,15] and storytelling [3], a point-of-view camera platform [6], and as an extension of the Snapchat mobile app [14,16].…”