RIT-Eyes: Rendering of near-eye images for eye-tracking applications

Nair, Nitinraj; Kothari, Rakshit; Chaudhary, Aayush K.; Yang, Zhizhuo; Díaz, Gabriel; Pelz, Jeff B.; Bailey, Reynold

doi:10.1145/3385955.3407935

Cited by 14 publications

(10 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…photo-realistic gaze and eye contact reconstruction Nair et al 2020;Schwartz et al 2020;Whitmire et al 2016;Wood et al 2016], eye editing for portrait images [Shu et al 2017], and retinal imaging [Huang et al 2014;Swedish et al 2015]. However, few works pay attention to eyelashes.…”

Section: Related Workmentioning

confidence: 99%

EyelashNet

Xiao

Zhang

et al. 2021

ACM Trans. Graph.

View full text Add to dashboard Cite

Eyelashes play a crucial part in the human facial structure and largely affect the facial attractiveness in modern cosmetic design. However, the appearance and structure of eyelashes can easily induce severe artifacts in high-fidelity multi-view 3D face reconstruction. Unfortunately it is highly challenging to remove eyelashes from portrait images using both traditional and learning-based matting methods due to the delicate nature of eyelashes and the lack of eyelash matting dataset. To this end, we present EyelashNet, the first eyelash matting dataset which contains 5,400 high-quality eyelash matting data captured from real world and 5,272 virtual eyelash matting data created by rendering avatars. Our work consists of a capture stage and an inference stage to automatically capture and annotate eyelashes instead of tedious manual efforts. The capture is based on a specifically-designed fluorescent labeling system. By coloring the eyelashes with a safe and invisible fluorescent substance, our system takes paired photos with colored and normal eyelashes by turning the equipped ultraviolet (UVA) flash on and off. We further correct the alignment between each pair of photos and use a novel alpha matte inference network to extract the eyelash alpha matte. As there is no prior eyelash dataset, we propose a progressive training strategy that progressively fuses captured eyelash data with virtual eyelash data to learn the latent semantics of real eyelashes. As a result, our method can accurately extract eyelash alpha mattes from fuzzy and self-shadow regions such as pupils, which is almost impossible by manual annotations. To validate the advantage of EyelashNet, we present a baseline method based on deep learning that achieves state-of-the-art eyelash matting performance with RGB portrait images as input. We also demonstrate that our work can largely benefit important real applications including high-fidelity personalized avatar and cosmetic design.

show abstract

Section: Related Workmentioning

confidence: 99%

EyelashNet

Xiao

Zhang

et al. 2021

ACM Trans. Graph.

View full text Add to dashboard Cite

show abstract

“…In general, approaches to eye segmentation [3,24,23,2,38,42] leverage deep convolutional neural networks [27,26,25] (CNNs) and, consequently, require large eye datasets in order to train these neural models effectively. The requisite datasets can be collected by recording synthetic information from simulations [35] or from human subjects directly [9,50]. Although, real-world eye datasets, such as OpenEDS [9] or MPIIGaze [50], provide invaluable samples of data/images to train the CNN models on, constructing such datasets requires a great deal of human annotation effort (high labeling burden) as well as introduces potential human subject image privacy issues.…”

Section: Introductionmentioning

confidence: 99%

“…Although, real-world eye datasets, such as OpenEDS [9] or MPIIGaze [50], provide invaluable samples of data/images to train the CNN models on, constructing such datasets requires a great deal of human annotation effort (high labeling burden) as well as introduces potential human subject image privacy issues. In contrast, synthetic eye datasets circumvent these issues, reducing the data collection effort inherent to working with actual human participants as well as manual labeling work needed to generate ground truth segmentation masks [35]. As a result, generating datasets of synthetic data samples offers the potential to train powerful eye-tracking CNN-based systems at greatly reduced overall cost.…”

Section: Introductionmentioning

confidence: 99%

Deep Domain Adaptation: A Sim2Real Neural Approach for Improving Eye-Tracking Systems

Nguyen,

Bailey,

Diaz

et al. 2024

Preprint

View full text Add to dashboard Cite

Eye image segmentation is a critical step in eye tracking that has great influence over the final gaze estimate. Segmentation models trained using supervised machine learning can excel at this task, their effectiveness is determined by the degree of overlap between the narrow distributions of image properties defined by the target dataset and highly specific training datasets, of which there are few. Attempts to broaden the distribution of existing eye image datasets through the inclusion of synthetic eye images have found that a model trained on synthetic images will often fail to generalize back to real-world eye images. In remedy, we use dimensionality-reduction techniques to measure the overlap between the target eye images and synthetic training data, and to prune the training dataset in a manner that maximizes distribution overlap. We demonstrate that our methods result in robust, improved performance when tackling the discrepancy between simulation and real-world data samples.

show abstract

“…Ray-tracing operations are then used to simulate the position of features used by an eye-tracker such as the pupil and glints of the user on the image sensor of a camera model. Contemporary simulation environments also include realistic head models which allow the simulation of synthetic images of the entire eye region ( 32 ; 19 , 22 ).…”

Section: Introductionmentioning

confidence: 99%

The influence of eye model parameter variations on simulated eye-tracking outcomes

Fischer,

Van den Heever,

Johan van der Merwe

2023

JEMR

View full text Add to dashboard Cite

The simulated data used in eye-tracking-related research has been largely generated using normative eye models with little consideration of how the variations in eye biometry found in the population may influence eye-tracking outcomes. This study investigated the influence that variations in eye model parameters have on the ability of simulated data to predict real-world eye-tracking outcomes. The real-world experiments performed by two pertinent comparative studies were replicated in a simulated environment using a high-complexity stochastic eye model that includes anatomically accurate distributions of eye biometry parameters. The outcomes showed that variations in anterior corneal asphericity significantly influence simulated eye-tracking outcomes of both interpolation and model-based gaze estimation algorithms. Other, more commonly varied parameters such as the corneal radius of curvature and foveal offset angle had little influence on simulated outcomes.

show abstract

RIT-Eyes: Rendering of near-eye images for eye-tracking applications

Cited by 14 publications

References 27 publications

EyelashNet

EyelashNet

Deep Domain Adaptation: A Sim2Real Neural Approach for Improving Eye-Tracking Systems

The influence of eye model parameter variations on simulated eye-tracking outcomes

Contact Info

Product

Resources

About