A deep learning approach for robust head pose independent eye movements recognition from videos

Abstract

Recognizing eye movements is important for gaze behavior understanding like in human communication analysis (human-human or robot interactions) or for diagnosis (medical, reading impairments). In this paper, we address this task using remote RGB-D sensors to analyze people behaving in natural conditions. This is very challenging given that such sensors have a normal sampling rate of 30 Hz and provide low-resolution eye images (typically 36x60 pixels), and natural scenarios introduce many variabilities in illumination, shadows, head pose, and dynamics. Hence gaze signals one can extract in these conditions have lower precision compared to dedicated IR eye trackers, rendering previous methods less appropriate for the task. To tackle these challenges, we propose a deep learning method that directly processes the eye image video streams to classify them into fixation, saccade, and blink classes, and allows to distinguish irrelevant noise (illumination, low-resolution artifact, inaccurate eye alignment, difficult eye shapes) from true eye motion signals. Experiments on natural 4-party interactions demonstrate the benefit of our approach compared to previous methods, including deep learning models applied to gaze outputs. CCS CONCEPTS • Computing methodologies → Tracking; Neural networks; Activity recognition and understanding.

Pupil-linked arousal correlates with neural activity prior to sensorimotor decisions

“…The method integrates denoising into segmentation and performs classification on the denoised segments into fixations, saccades, smooth pursuits, and post-saccadic oscillations. It has previously been applied to noisy data to recover gaze position, and velocity estimates in experiments with complex gaze behavior [72,73]. Although the method estimates the signal's noise level and determines gaze feature parameters from human classification examples in a data-driven manner, our empirical testing showed that performance was drastically improved if the estimation was performed using at least 3 s of data.…”

CLERA: A Unified Model for Joint Cognitive Load and Eye Region Analysis in the Wild

“…In recent years, deep learning methods have been used to form better representations in order to improve the accuracy and robustness of general object detection [32,60,63] and keypoint detection [6,8,32,54]. Some relevant papers [2,10,40,65] explore using existing deep learning architectures on the tasks of blink or gaze estimation. Our work takes a further step to propose a method for precise keypoint detection and a unified framework designed specifically for joint eye, pupil, and blink detection.…”

RT-BENE: A Dataset and Baselines for Real-Time Blink Estimation in Natural Environments

“…To the best of our knowledge, there are no deep learning based methods that jointly consider blink estimation and gaze estimation. However, note that Siegfried et al [41] recently proposed a deep learning based method to classify gaze streams into fixation, saccade, and blink classes.…”