Coarse-to-Fine Image Inpainting via Region-wise Convolutions and Non-Local Correlation

Abstract

Recently deep neural networks have achieved promising performance for filling large missing regions in image inpainting tasks. They usually adopted the standard convolutional architecture over the corrupted image, where the same convolution filters try to restore the diverse information on both existing and missing regions, and meanwhile ignores the long-distance correlation among the regions. Only relying on the surrounding areas inevitably leads to meaningless contents and artifacts, such as color discrepancy and blur. To address these problems, we first propose region-wise convolutions to locally deal with the different types of regions, which can help exactly reconstruct existing regions and roughly infer the missing ones from existing regions at the same time. Then, a non-local operation is introduced to globally model the correlation among different regions, promising visual consistency between missing and existing regions. Finally, we integrate the region-wise convolutions and non-local correlation in a coarse-to-fine framework to restore semantically reasonable and visually realistic images. Extensive experiments on three widely-used datasets for image inpainting tasks have been conducted, and both qualitative and quantitative experimental results demonstrate that the proposed model significantly outperforms the state-of-the-art approaches, especially for the large irregular missing regions.

Semantic-SCA: Semantic Structure Image Inpainting With the Spatial-Channel Attention

“…To make full use of the mask information, different researchers propose different novel convolution methods [8], [27], [28]. Liu et al [8] propose a partial convolution to distinguish the effective area of the original image.…”

“…Liu et al [8] propose a partial convolution to distinguish the effective area of the original image. Ma et al [28] propose region-wise convolutions to deal with effective and ineffective regions. To further extract the information from the original image, those methods [29]- [31] introduce the multi-scale mechanism.…”

Progressively Inpainting Images Based on a Forked-Then-Fused Decoder Network

“…The ill-posedness of image inpainting can be distilled into the following: how to seek the most proper hypothesis for the corrupted region conditioned on the valid surroundings. In the past decade, researchers have devoted substantial efforts to this field, which can be mainly divided into three categories: diffusion-based methods [ 17 , 18 , 19 , 20 ], patch-based methods [ 21 , 22 , 23 , 24 , 25 , 26 ], and CNN (Convolutional Neural Network)-based methods [ 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 ].…”

“…More interestingly, some exquisite networks, such as GAN (Generative Adversarial Network) [ 51 ] or VAE (Variational Auto-Encoder) [ 52 ], excel at creating realistic samples. Thus, the CNN-based methods [ 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 ] have been a recent surge of interests in the field of image inpainting. Pathak et al [ 27 ] set up a CE (Context Encoder) network that is of a channel-wise fully connected layer in the middle.…”

“…However, although these methods have the capability to create novel contents, there still exist semantic faults or visual artifacts such as color discrepancy and texture distortion. To address these problems, advanced inpainting mechanisms in the following three aspects: specialized convolution operations [ 30 , 31 , 32 , 33 ], contextual attention module [ 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 ], and progressive inpainting strategies [ 46 , 47 , 48 , 49 , 50 ], have been studied recently.…”