Cross-Modal Face Matching: Beyond Viewed Sketches

Abstract

Abstract. Matching face images across different modalities is a challenging open problem for various reasons, notably feature heterogeneity, and particularly in the case of sketch recognition -abstraction, exaggeration and distortion. Existing studies have attempted to address this task by engineering invariant features, or learning a common subspace between the modalities. In this paper, we take a different approach and explore learning a mid-level representation within each domain that allows faces in each modality to be compared in a domain invariant way. In particular, we investigate sketch-photo face matching and go beyond the well-studied viewed sketches to tackle forensic sketches and caricatures where representations are often symbolic. We approach this by learning a facial attribute model independently in each domain that represents faces in terms of semantic properties. This representation is thus more invariant to heterogeneity, distortions and robust to mis-alignment. Our intermediate level attribute representation is then integrated synergistically with the original low-level features using CCA. Our framework shows impressive results on cross-modal matching tasks using forensic sketches, and even more challenging caricature sketches. Furthermore, we create a new dataset with ≈ 59, 000 attribute annotations for evaluation and to facilitate future research.

Attribute-Enhanced Face Recognition with Neural Tensor Fusion Networks

“…Unlike linear methods based on averaging, concatenation, linear subspace learning [8,27], or LDA [3], our fusion method is non-linear, which is more powerful to model complex problems. Furthermore, comparing with other first-order non-linear methods based on element-wise combinations only [28], our method is higher order: it accounts for all interactions between each pair of feature channels in both views.…”

“…Detected facial attributes can be applied directly to authentication. Facial attributes have been applied to enhance face verification, primarily in the case of cross-modal matching, by filtering [19,54] (requiring potential FRF matches to have the correct gender, for example), model switching [18], or aggregation with conventional features [27,17]. [21] defines 65 facial attributes and proposes binary attribute classifiers to predict their presence or absence.…”

“…For example, the fusion method can be trained with losses known to be effective for face representation learning: identity-supervised softmax, and centre-loss [47]. Alternative methods are either unsupervised [8,27], constrained in the types of supervision they can exploit [3,17], or only stack scores rather than improving a learned representation [48,37]. Therefore, they are relatively ineffective at learning how to combine the two-source information in a task-specific way.…”

Fine-grained sketch-based image retrieval: The role of part-aware attributes

“…Baselines (fine-grained SBIR): We evaluate against when singular feature representations are used: (i) Part-HOG, where part-level HOG is employed, (ii) Part-Attribute, where only automatically detected partaware attributes are utilized, and (iii) Part-Structure, where geometric part structure alone is used to retrieve. We also adopt the state-of-the-art two-view CCA method previously utilized to match facial sketches and caricatures to mugshot photo [25]. We compare with three pair-wise configurations to accommodate their two-view setting: Part-HOG+Part-Attribute+2View-CCA, Part-HOG+Part-Structure+2View-CCA and Table 1.…”

Deep Sketch-Photo Face Recognition Assisted by Facial Attributes

“…They also employed some soft biometric traits such as gender, ethnicity, and skin color to reorder the ranked list of the suspects. Ouyang et al [27] introduced a framework to combine the facial attributes with low-level features to fill the gap between sketch and photo modalities.…”