Shift-Net: Image Inpainting via Deep Feature Rearrangement

Yan, Zhaoyi; Li, Xiaoming; Li, Mu; Zuo, Wangmeng; Shan, Shiguang

doi:10.1007/978-3-030-01264-9_1

Cited by 426 publications

(368 citation statements)

References 49 publications

Supporting

Mentioning

363

Contrasting

Unclassified

Order By: Relevance

“…It is obvious that our model is superior to current state-of-the-art methods on multiple datasets. More quantitative comparisons of realism with CA [20], SH [17], and GL [39] on the CelebA [33], AgriculturalDisease, and MauFlex [1] datasets were also conducted. Table 4 and Table 5 list the evaluation results on the AgriculturalDisease and MauFlex datasets, respectively.…”

Section: A More Comparisons Resultsmentioning

confidence: 99%

“…Furthermore, our inpainting network which is implemented in TensorFlow [41] To fairly evaluate our method, we only conducted experiments on the centering hole. We compared our method with GL [39], CA [20], and SH [17] on images from the CelebA [33], Agricultural Disease, and MauFlex [1] validation sets. The size of all mask images were processed to 128 × 128 for training and testing.…”

Section: Implementation Detailsmentioning

confidence: 99%

“…It is not difficult to see that our methods are superior to all other methods in terms of these quantitative tests. Table 1: Results using the CelebA dataset with large missing regions, comparing global and local (GL) [39], Shift-net (SH) [17], Contextual Attention (CA) [20], and ours method. − Lower is better.…”

Section: Quantitative Comparisonsmentioning

confidence: 99%

“…CA Contextual Attention, proposed by Yu et al[20] -SH Shift-net, proposed by Yan et al[17] -GL Global and local, proposed by Iizuka et al[39] …”

mentioning

confidence: 99%

“…Results using the AgriculturalDisease dataset with large missing regions, comparing GL[39], SH[17], CA[20], and ours method. − Lower is better.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Diversity-Generated Image Inpainting with Style Extraction

Cai¹,

Wei²

2019

Preprint

View full text Add to dashboard Cite

The latest methods based on deep learning have achieved amazing results regarding the complex work of inpainting large missing areas in an image. This type of method generally attempts to generate one single "optimal" inpainting result, ignoring many other plausible results. However, considering the uncertainty of the inpainting task, one sole result can hardly be regarded as a desired regeneration of the missing area. In view of this weakness, which is related to the design of the previous algorithms, we propose a novel deep generative model equipped with a brand new style extractor which can extract the style noise (a latent vector) from the ground truth image. Once obtained, the extracted style noise and the ground truth image are both input into the generator. We also craft a consistency loss that guides the generated image to approximate the ground truth. Meanwhile, the same extractor captures the style noise from the generated image, which is forced to approach the input noise according to the consistency loss. After iterations, our generator is able to learn the styles corresponding to multiple sets of noise. The proposed model can generate a (sufficiently large) number of inpainting results consistent with the context semantics of the image. Moreover, we check the effectiveness of our model on three databases, i.e., CelebA, Agricultural Disease, and MauFlex. Compared to state-of-the-art inpainting methods, this model is able to offer desirable inpainting results with both a better quality and higher diversity, for example, on the human face, we can even output the different gaze angles of the eyes and whether they have glasses, etc. The code and model will be made available on https://github.com/vivitsai/SEGAN.

show abstract

Section: A More Comparisons Resultsmentioning

confidence: 99%

Section: Implementation Detailsmentioning

confidence: 99%

Section: Quantitative Comparisonsmentioning

confidence: 99%

“…CA Contextual Attention, proposed by Yu et al[20] -SH Shift-net, proposed by Yan et al[17] -GL Global and local, proposed by Iizuka et al[39] …”

mentioning

confidence: 99%

“…Results using the AgriculturalDisease dataset with large missing regions, comparing GL[39], SH[17], CA[20], and ours method. − Lower is better.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Diversity-Generated Image Inpainting with Style Extraction

Cai¹,

Wei²

2019

Preprint

View full text Add to dashboard Cite

show abstract

PIPformers: Patch based inpainting with vision transformers for generalize paintings

Lee,

Kang

2024

Computer Animation & Virtual

View full text Add to dashboard Cite

Image inpainting is a field that has been traditionally attempted in the field of computer vision. After the development of deep learning, image inpainting has been advancing endlessly together with convolutional neural networks and generative adversarial networks. Thereafter, it has been expanded to various fields such as image filing through guiding and image inpainting using various masking. Furthermore, the field termed image out‐painting has also been pioneered. Meanwhile, after the recent announcement of the vision transformer, various computer vision problems have been attempted using the vision transformer. In this paper, we are trying to solve the problem of image generalization painting using the vision transformer. This is an attempt to fill images with painting no matter whether the areas where painting is missing are in or out of the images, and without guiding. To that end, the painting problem was defined as a problem to drop images in patch units for easy use in the vision transformer. And we solved the problem with a simple network structure created by slightly modifying the vision transformer to fit the problem. We named this network PIPformers. PIPformers achieved better values than other papers compared to PSNR, RMSE and SSIM.

show abstract

PKD‐Net: Distillation of prior knowledge for image completion by multi‐level semantic attention

Lu,

Lin,

Xing

et al. 2023

Concurrency and Computation

View full text Add to dashboard Cite

SummaryPrior knowledge plays a crucial role in image completion. Although almost all of the existing image completion methods use prior knowledge to complete the image to be repaired from different perspectives, the learning and modeling of the prior knowledge is still a challenging problem. In order to address this issue, we propose a novel prior knowledge distillation framework (PKD‐Net) which could distill prior knowledge of structure and style from multiple semantic space and generates not only plausible content but also consistent style with surrounding image area. Our PKD‐Net replaces the skip connection in the vanilla U‐Net with a semantic shift attention module. The semantic shift attention module takes features from encoder layer and those from decoder layer as input pairs to output shifted features which take into account the long‐range dependency of encoder layer features and corresponding decoder layer features from the perspective of local structure and style. Semantic shift attention module models the global interdependencies in local spatial structures (patches centered at each position) and style (appearance texture) dimensions respectively, which could implement distillation of prior knowledge from two aspects: structure and style. Experiments on multiple datasets including faces (CelebA, CelebA‐HQ) and natural images (ImageNet, Places2, Paris Street View) demonstrate that our proposed approach generates higher quality completion results than existing ones.

show abstract

Shift-Net: Image Inpainting via Deep Feature Rearrangement

Cited by 426 publications

References 49 publications

Diversity-Generated Image Inpainting with Style Extraction

Diversity-Generated Image Inpainting with Style Extraction

PIPformers: Patch based inpainting with vision transformers for generalize paintings

PKD‐Net: Distillation of prior knowledge for image completion by multi‐level semantic attention

Contact Info

Product

Resources

About