R-MNet: A Perceptual Adversarial Network for Image Inpainting

Abstract

Facial image inpainting is a problem that is widely studied, and in recent years the introduction of Generative Adversarial Networks, has led to improvements in the field. Unfortunately some issues persists, in particular when blending the missing pixels with the visible ones. We address the problem by proposing a Wasserstein GAN combined with a new reverse mask operator, namely Reverse Masking Network (R-MNet), a perceptual adversarial network for image inpainting. The reverse mask operator transfers the reverse masked image to the end of the encoder-decoder network leaving only valid pixels to be inpainted. Additionally, we propose a new loss function computed in feature space to target only valid pixels combined with adversarial training. These then capture data distributions and generate images similar to those in the training data with achieved realism (realistic and coherent) on the output images. We evaluate our method on publicly available dataset, and compare with state-of-the-art methods. We show that our method is able to generalize to highresolution inpainting task, and further show more realistic outputs that are plausible to the human visual system when compared with the state-of-the-art methods.

V-LinkNet: Learning Contextual Inpainting Across Latent Space of Generative Adversarial Network

“…GAN models for image inpainting [20,33,34] have used multicolumns to encode and propagated features directly to the decoder or use a selfsupervised Siamese style inference approach [21], where a style encoder is the supervisor of the generator, to improve feature extraction and learning. Other methods [35,25,18] observed that failures in feature extraction and propagation could be due to the irregular holes. To address the limitation, Liu et al [11] proposed an independent mask updating with partial convolutions to specifically target missing regions.…”

“…Yu et al [25] proposed to use gated convolutions to gear the model towards learning soft mask of the irregular hole regions. More recently, Jam et al [18] proposed a reverse mask mechanism to specifically target missing regions whilst preserving the visible ones using a spatial preserving operation.…”

“…where, K is a constant, is the element-wise multiplication, I gt is the groundtruth image and I pred is the predicted image. Further, we utilize the reversed mask loss (L rm ) from [18] and compute a contextual loss. We want to keep the known pixel locations of the input image by penalizing the predictions thus creating similar pixels based on the reversed mask and masked regions.…”

“…The learning-based methods, popular known as deep generative neural networks, have become the state of the art, based on their ability to learn distribution with regards to context. These approaches [9,10,11,12,13,14,15,16,17,18,19] use convolutional neural network (CNN) within an encoder-decoder within a GAN-based network to generate realistic images. These algorithms with a wide range of parameters and layers learn to manage feature extraction, propagation, and regularisation.…”

“…It is obvious that GAN-based inpainting methods [9,11,25,18,16,19] have made tremendous strides in terms of visual quality, as measured by the popular Structure Similarity Index Measure (SSIM) [26] and other quantitative evaluations [27,28,29,30]. However, the achieved advancement, which is always judged over existing models by the aforementioned measures, is not as significant as projected.…”