3D Gaze Estimation Using RGB-IR Cameras

Abstract

In this paper, we present a framework for 3D gaze estimation intended to identify the user’s focus of attention in a corneal imaging system. The framework uses a headset that consists of three cameras, a scene camera and two eye cameras: an IR camera and an RGB camera. The IR camera is used to continuously and reliably track the pupil and the RGB camera is used to acquire corneal images of the same eye. Deep learning algorithms are trained to detect the pupil in IR and RGB images and to compute a per user 3D model of the eye in real time. Once the 3D model is built, the 3D gaze direction is computed starting from the eyeball center and passing through the pupil center to the outside world. This model can also be used to transform the pupil position detected in the IR image into its corresponding position in the RGB image and to detect the gaze direction in the corneal image. This technique circumvents the problem of pupil detection in RGB images, which is especially difficult and unreliable when the scene is reflected in the corneal images. In our approach, the auto-calibration process is transparent and unobtrusive. Users do not have to be instructed to look at specific objects to calibrate the eye tracker. They need only to act and gaze normally. The framework was evaluated in a user study in realistic settings and the results are promising. It achieved a very low 3D gaze error (2.12) and very high accuracy in acquiring corneal images (intersection over union—IoU = 0.71). The framework may be used in a variety of real-world mobile scenarios (indoors, indoors near windows and outdoors) with high accuracy.

Calibration-Free Mobile Eye-Tracking Using Corneal Imaging

“…The present work extends our previous work [ 15 ]. Here, we use the device offered in [ 15 ], along with the computed 3D model and the 3D-mapping transformation between IR and RGB eye cameras, integrating the per-user point-to-point transformation from the RGB eye camera to the front-scene camera to build a complete calibration-free mobile eye-tracking system. These system components (modeling, 3D-mapping transformation, and integration) and the system as a whole comprise the main contribution of this paper.…”

“…The paper by Mokatren et al [ 15 ] introduces a method for 3D gaze estimation through RGB-IR cameras, eliminating the requirement for calibration. The hardware setup involves a headset equipped with two cameras—an IR camera for pupil detection and an RGB camera for capturing corneal images.…”

“…However, in our specific scenario, we employ the eyeball model, which enables us to transform a point directly into another point. The proposed 3D-mapping transformation, as detailed in [ 15 ], precisely accomplishes this task. Our approach relies solely on the 3D-mapping transformation between the IR and RGB cameras.…”

“…Since this is challenging, as the detection is affected by lighting conditions, especially outdoors and in harsh lighting conditions, we detect the pupil using the IR eye camera. The camera provides a reliable image of the pupil whose bounding box is then transformed to the RGB eye camera’s image plane using the 3D-mapping transformation [ 15 ] (see Figure 7 , left). We can extract the corneal image by cropping the RGB eye image around the pupil using the transformed bounding box.…”

“…To eliminate the cumbersome calibration procedure, we explored a new method for gaze estimation which is based on corneal imaging [ 14 ]. The system we developed uses three cameras, a scene camera and two eye cameras: an IR camera and an RGB camera [ 15 ], which enable a 3D perspective. To understand corneal imaging, we must remember that the visible parts of the human eye are the white sclera continuous with the cornea, the iris, and the black pupil, which are all protected by tear fluid.…”