Interactive Removal of Microphone Object in Facial Images

Abstract

Removing a specific object from an image and replacing the hole left behind with visually plausible backgrounds is a very intriguing task. While recent deep learning based object removal methods have shown promising results on this task for some structured scenes, none of them have addressed the problem of object removal in facial images. The objective of this work is to remove microphone object in facial images and fill hole with correct facial semantics and fine details. To make our solution practically useful, we present an interactive method called MRGAN, where the user roughly provides the microphone region. For filling the hole, we employ a Generative Adversarial Network based image-to-image translation approach. We break the problem into two stages: inpainter and refiner. The inpainter estimates coarse prediction by roughly filling in the microphone region followed by the refiner which produces fine details under the microphone region. We unite perceptual loss, reconstruction loss and adversarial loss as joint loss function for generating a realistic face and similar structure to the ground truth. Because facial image pairs with or without microphone do not exist, we have trained our method on a synthetically generated microphone dataset from CelebA face images and evaluated on real world microphone images. Our extensive evaluation shows that MRGAN performs better than state-of-the-art image manipulation methods on real microphone images although we only train our method using the synthetic dataset created. Additionally, we provide ablation studies for the integrated loss function and for different network arrangements.

A Novel GAN-Based Network for Unmasking of Masked Face

“…In this section, we analyze results generated by our model and compare with other state-of-the art image editing methods such as Iizuka et al [6], Yuet al [9], EdgeConnect [11] and MRGAN [8] both quantitatively and qualitatively on real world test images. Figure 5 shows the sample generated by our model for real test images.…”

“…For object removal, Contextual Attention (GCA) [9] and MRGAN [8] use a two stage network. The first stage network produces a coarse result while the second stage network refines the output from the first stage.…”

“…These schemes do not work for our problem because most of the time the first stage is unable to generate a reasonable map due to the large size of missing region. Moreover, all these deep learning based image editing works [6], [8], [9], [11], [22] assume that users provide the object map at the inference stage. In [16], they automatically detect object region and remove it in general scene-level images.…”

“…For fair comparison, we have trained current stateof-the-art approaches Iizuka et al [6], Yu et al [9], EdgeConnect [11] and MRGAN [8] using our synthetic dataset. We also provide the object binary map generated by our map module along with input image both at training and inference stages because all these methods assume that object binary map is given.…”

“…In the first stage, they generate a coarse output and refine it in the second stage. Khan et al [8] also employ a coarseto-fine network approach to remove microphone object from facial images. SPG-Net [12] and EdgeConnect [11] also use a two-staged adversarial approach to remove unwanted object.…”