EyeGAN: Gaze–Preserving, Mask–Mediated Eye Image Synthesis

Abstract

Automatic synthesis of realistic eye images with prescribed gaze direction is important for multiple application domains. We introduce EyeGAN, an algorithm to generate eye images in the style of a desired target domain, that inherit annotations available in images from a source domain. EyeGAN takes in input ternary masks, which are used as domain-independent proxies for gaze direction. We evaluate EyeGAN against competing eye image synthesis algorithms by measuring a specific gaze consistency index. In addition, we present results from multiple experiments (involving eye region segmentation, pupil localization, and gaze direction estimation) showing that the use of EyeGANgenerated images with inherited annotations for network training leads to superior performances compared to other domain transfer algorithms.

Subject Guided Eye Image Synthesis with Application to Gaze Redirection

“…When paired data is not available, unsupervised methods can be used [39], [28], [16] , [23]. While SimGAN [28] and CycleGAN [21] translate synthetic eye images into real world images, EyeGAN [15] starts from ternary eye segmentation masks. EyeGAN is trained using pix2pix [12] on image/segmentation input pairs, where the segmentation mask is extracted from the input image.…”

“…Here, E g1 s and E g2 s are images of the same individual with different gaze directions. M (•) is a function that extracts a ternary mask from an eye image [15]. Synthesis Loss.…”

“…As a consequence, models trained on these images may not do well when applied to real-world eye images. To overcome this problem, some authors have proposed methods for synthesizing real-world eye images from synthetic eye images while maintaining annotations (e.g, gaze direction; [28], [21], [15].) While effective, these eye image synthesis methods do not give the user much control over the generated eye features -skin color, texture, iris color, and eye shape.…”

“…In this context, the content to be manipulated is gaze direction. Following Kaur et al [15], we use ternary segmentation masks to characterize gaze direction. Masks act as style-independent proxies for gaze.…”

“…Masks act as style-independent proxies for gaze. Kaur et al [15] introduced an algorithm (EyeGAN) that takes a synthetic mask for a prescribed gaze direction as input, and generates an image under content (gaze direction) consistent with the input mask and some random style. In this work, we push this idea forth, and introduce a new cyclic mechanism to ensure consistency of both style and content of the generated image.…”