Generative Facial Prior and Semantic Guidance for Iterative Face Inpainting

Abstract

Image inpainting techniques have been greatly improved by relying on structure and texture priors. However, damaged original images or rough predictions cannot provide sufficient texture information and accurate structural priors, leading to a drop in image quality. Moreover, from the perspective of human eye perception, it is important to pay attention to facial symmetry and facial attribute consistency. In this paper, we present a face inpainting system with iteration structure, which uses generative facial priors contained in pretrained GANs and predicted semantic information for guidance. Specifically, generative facial priors generated by the GAN inversion techniques introduce sufficient textures and features to assist inpainting; semantic maps are able to provide facial structural information and semantic categories of different pixels for face reconstruction. In particular, we iteratively refine images multiple times, with the semantic map updated in each iteration. The Weighted Prior-Guidance Modulation layer (WPGM) is devised for incorporating priors into networks through spatial modulation. We also propose facial feature self-symmetry loss to restrict the symmetry of faces in feature space. Experiments on CelebA-HQ and LaPa datasets demonstrate the superiority of our model for facial detail and attributes consistency. Meanwhile, under the background of COVID-19, it is worth trying recognition via inpainting to deal with recognition challenges brought by mask occlusion. Relevant experiments show that our inpainting model does help to recognition tasks to a certain degree, with higher accuracy.