RIT-Eyes: realistically rendered eye images for eye-tracking applications

Nair, Nitinraj; Chaudhary, Aayush K.; Kothari, Rakshit; Díaz, Gabriel; Pelz, Jeff B.; Bailey, Reynold

doi:10.1145/3379157.3391990

Cited by 7 publications

(1 citation statement)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We purchased 48 human models which retopologised from real-world human 3D scanning from an online repository, following the synthetic eye tracking paper, RIT eyes. 12 These 48 human models include people of different ages, genders, and ethnicity. The purchased human models include RGB texture maps, but we manually adjusted the reflectance of the textures according to our real-world experiment, so that the average DC and AC levels of the sinusoidal signals in the rendered images are close to the ones in the real-world experiment.…”

Section: Headmentioning

confidence: 99%

Fringe projection-based single-shot 3D eye tracking using deep learning and computer graphics

Zheng¹,

Chao²,

An³

et al. 2023

Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) IV

View full text Add to dashboard Cite

High-accuracy and high-speed 3D sensing technology plays an essential role in VR eye tracking as it can build a bridge to connect the user with virtual worlds. In VR eye tracking, fringe projection profilometry (FPP) can avoid dependence on scene textures and provide accurate results in near-eye scenarios; however, phase-shifting based FPP faces challenges like motion artifacts and may not meet the low-latency requirements of eye tracking tasks. On the other hand, Fourier transform profilometry can achieve single-shot 3D sensing, but the system is highly impacted by the texture variations on the eye. As a solution to the challenges above, researchers have explored deep learning-based single-shot fringe projection 3D sensing techniques. However, building a training dataset is expensive, and without abundant data the model is difficult to make generalized. In this paper, we built a virtual fringe projection system along with photorealistic face and eye models to synthesize large amounts of training data. Therefore, we can reduce the cost and enhance the generalization ability of the convolutional neural network (CNN). The training data synthesizer utilizes physically based rendering (PBR) and achieves high photorealism. We demonstrate that PBR can simulate the complex double refraction of structured light due to corneas. To train the CNN, we adopted the idea of transfer learning, where the CNN is first trained by PBR-generated data, then trained with the real data. We tested the CNN on real data, and the predicted results demonstrate that the synthesized data enhances the performance of the model and achieves around 3.722 degree gaze accuracy and 0.5363 mm pupil position error on an unfamiliar participant.

show abstract

Section: Headmentioning

confidence: 99%