Disentangled Face Identity Representations for joint 3D Face Recognition and Expression Neutralisation

Abstract

In this paper, we propose a new deep learning based approach for disentangling face identity representations from expressive 3D faces. Given a 3D face, our approach not only extracts a disentangled identity representation, but also generates a realistic 3D face with a neutral expression while predicting its identity. The proposed network consists of three components; (1) a Graph Convolutional Autoencoder (GCA) to encode the 3D faces into latent representations, (2) a Generative Adversarial Network (GAN) that translates the latent representations of expressive faces into those of neutral faces, (3) and an identity recognition sub-network taking advantage of the neutralized latent representations for 3D face recognition. The whole network is trained in an end-to-end manner. Experiments are conducted on three publicly available datasets showing the effectiveness of the proposed approach.

Adversarial 3D Face Disentanglement of Identity and Expression

“…This is different from [41] who use a discriminator to enforce independence of two distributions, which is based on Kim and Mnih's PMLR 2018 work [19]. Our work is also different from [17], who use a discriminator in the latent space to learn a valid translation from expressive to neutral representations. The "Ours" in these tables means that our method does not access the neutral ground-truths in end-to-end training, which fits to some real-world scenarios where corresponding identity shapes are not available.…”

“…Zhang et al [41] combined a VAE with an adversarial network in order to eliminate correlations between identity and expression representations and ensure their independence. Kacem et al [17] employed a GAN to extract expressive representations. Zhang et al [40] modelled expressions as the deviation from the identity and extracted a deviation feature vector using a deviation learning network with a pseudo-siamese structure.…”

“…We adopt the evaluation metrics used in [16], [17], [41], i.e. both reconstruction and disentanglement metrics.…”

“…We compare our work with five state-of-the-art 3D face disentanglement methods, FLAME [24], Jiang et al [16], and DI-MeshEncoder [41] on CoMA and BU3DFE, and Kacem et al [17] and Convolutional Mesh Autoencoder (Conv-MeshAE) [32] on FaceScape. We carefully report results from [41] Although this is a 3D reconstruction method instead of disentangling identity and expression shapes, we follow [17] to consider pairs of expressive and neutral faces as its input and ground-truths, respectively. We use a widely-adopted autoencoder structure based on PointNet that is unlike these four methods and, furthermore, neutral ground truths are not required in our method.…”

FaceTuneGAN: Face Autoencoder for Convolutional Expression Transfer Using Neural Generative Adversarial Networks

“…On the CoMA dataset, both our content and style spaces have 4 dimensions, adding up to a total dimensionality of 8. On the FaceScape dataset, Kacem et al [23], use a single latent space of 25 dimensions. However, this latent space is only used to represent neutral faces.…”