Cross-Spectral Face Hallucination via Disentangling Independent Factors

Abstract

The cross-sensor gap is one of the challenges that arise much research interests in Heterogeneous Face Recognition (HFR). Although recent methods have attempted to fill the gap with deep generative networks, most of them suffered from the inevitable misalignment between different face modalities. Instead of imaging sensors, the misalignment primarily results from geometric variations (e.g., pose and expression) on faces that stay independent from spectrum. Rather than building a monolithic but complex structure, this paper proposes a Pose Agnostic Cross-spectral Hallucination (PACH) approach to disentangle the independent factors and deal with them in individual stages. In the first stage, an Unsupervised Face Alignment (UFA) network is designed to align the nearinfrared (NIR) and visible (VIS) images in a generative way, where 3D information is effectively utilized as the pose guidance. Thus the task of the second stage becomes spectrum transform with paired data. We develop a Texture Prior Synthesis (TPS) network to accomplish complexion control and consequently generate more realistic VIS images than existing methods. Experiments on three challenging NIR-VIS datasets verify the effectiveness of our approach in producing visually appealing images and achieving state-of-the-art performance in cross-spectral HFR.

High Fidelity Face Manipulation with Extreme Poses and Expressions

“…Other face manipulation tasks also have made considerable development, such as facial makeup [32], age synthesis [26], [33] face inpainting [34], [35], cross spectral synthesis [36]. PairedCycleGAN [32] introduces a new cycle generative network that transfers makeup styles and removes styles in an asymmetric manner.…”

“…[34] utilizes the surrounding background patches to facilitate image inpainting. [36] introduces a pose alignment module and a texture prior generator to tackle the unpaired cross spectral synthesis problem.…”