Prepended Domain Transformer: Heterogeneous Face Recognition Without Bells and Whistles

Abstract

Heterogeneous Face Recognition (HFR) refers to matching face images captured in different domains, such as thermal to visible images (VIS), sketches to visible images, near-infrared to visible, and so on. This is particularly useful in matching visible spectrum images to images captured from other modalities. Though highly useful, HFR is challenging because of the domain gap between the source and target domain. Often, large-scale paired heterogeneous face image datasets are absent, preventing training models specifically for the heterogeneous task. In this work, we propose a surprisingly simple, yet, very effective method for matching face images across different sensing modalities. The core idea of the proposed approach is to add a novel neural network block called Prepended Domain Transformer (PDT) in front of a pre-trained face recognition (FR) model to address the domain gap. Retraining this new block with few paired samples in a contrastive learning setup was enough to achieve stateof-the-art performance in many HFR benchmarks. The PDT blocks can be retrained for several source-target combinations using the proposed general framework. The proposed approach is architecture agnostic, meaning they can be added to any pre-trained FR models. Further, the approach is modular and the new block can be trained with a minimal set of paired samples, making it much easier for practical deployment. The source code and protocols will be made available publicly.

One-way multimodal image-to-image translation for heterogeneous face recognition

“…However, these methods did not truly address the lack of heterogeneous data problem and cannot create large-scale VIS–NIR paired face data. Recently, RDT 36 proposed a neural network block called the pre-domain transformer to address the domain gap issue, which is added before the pre-trained face recognition model. DVG-Face 37 proposes a dual variational generator to learn the joint distribution of paired heterogeneous images and integrates rich identity information from large-scale visible data into the joint distribution.…”

HyperFace: A Deep Fusion Model for Hyperspectral Face Recognition

“…Last but not least, IR (especially short-wave IR) is used to sense the environment at night and under harsh atmospheric conditions such as rain, snow, and fog [9,10], making it a reliable imaging alternative to visible light in many real-world scenarios such as military and law enforcement applications. Dealing with different IR bands, by either stating the face recognition problem as an intra-spectral or a cross-spectral problem, has attracted the attention of several research groups [11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27]. Intra-spectral IR face recognition usually considers a single band within the IR spectral range and matches IR face images against other IR face images within the same band [12].…”