Image inpainting using frequency-domain priors

Roy, Hiya; Chaudhury, Subhajit; Yamasaki, Toshihiko; Hashimoto, Tatsuaki

doi:10.1117/1.jei.30.2.023016

Cited by 16 publications

(10 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since the image inpainting algorithm [4] that uses both frequency domain and spatial domain information has a good performance on the public image inpainting dataset. Therefore, we construct the network model by copying and improving this algorithm.…”

Section: Proposed Methodsmentioning

confidence: 99%

“…The VMC module dynamically selects sampling locations based on the information in feature maps for flexible learning and the RCN module can adaptively learn weights. In addition, there are algorithms that combines frequency domain and spatial domain [4][5][6][7][8], which propose to use DFT or DWT to model frequency domain information, they can restore clearer texture details. Therefore, in this paper, we also use the combination of frequency domain and spatial domain to realize the network.…”

Section: Related Workmentioning

confidence: 99%

“…However, the inpainting effect on the textured images is still not ideal to date. Some algorithms adopted the combination of frequency and spatial domains to inpaint textured images [4][5][6][7][8]. The frequency domain images, as obtained by discrete fourier transform (DFT) or discrete wavelet transform (DWT), can be used as information for network training.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A deep learning image inpainting method based on stationary wavelet transform

Huang,

Lu,

Chen

et al. 2023

Multimedia Systems

View full text Add to dashboard Cite

The development of deep learning has greatly improved the image inpainting performance in the past decades. To inpaint images with specific tasks usually require different network models. For instance, the highly structured images need to recover the structural consistency, and the textured ones restore the local high-frequency details. However, it is still challenging to realize an effective algorithm taking account of the global structure and texture details separately. Herein, we proposed a two-stage inpainting method, by combining the information of frequency and spatial domain networks. Stationary wavelet transform (SWT) with good time-frequency characteristics was applied to obtain the sub-band images as the basic input for frequency domain inpainting. Contextual attention layer (CAL) modules were optionally introduced in the network model to adapt to various inpainting tasks. We also tested and discussed the impacts of some commonly used loss functions, including normal L1 loss, normal GAN loss, weighted L1 loss and WGAN-GP loss, on highly structured and textured images.

show abstract

Section: Proposed Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A deep learning image inpainting method based on stationary wavelet transform

Huang,

Lu,

Chen

et al. 2023

Multimedia Systems

View full text Add to dashboard Cite

show abstract

“…Considering both structure and texture reconstruction, Liu et al [37] design a mutual encoder-decoder based framework to improve the quality of synthesized content. Roy et al [40] present an image inpainting method using frequency-domain information as structural priors to reconstruct the intact image. Feng et al [41] introduce a deep-masking mechanism to infer corrupted areas from the known content in a coarse-to-fine manner.…”

Section: B Deep Learning-based Methods For Image Inpaintingmentioning

confidence: 99%

Collaborative Contrastive Learning-Based Generative Model for Image Inpainting

Liu

Chen

2022

IEEE Access

View full text Add to dashboard Cite

The critical challenge of image inpainting is to infer reasonable semantics and textures for a corrupted image. Typical methods for image inpainting are built upon some prior knowledge to synthesize the complete image. One potential limitation is that those methods often remain undesired blurriness or semantic mistakes in the synthesized image while handling images with large corrupted areas. In this paper, we propose a Collaborative Contrastive Learning-based Generative Model (C2LGM), which learns the content consistency in the same image to ensure that the inferred content of corrupted areas is reasonable compared to the known content by pixel-level reconstruction and high-level semantic reasoning. C2LGM leverages the encoder-decoder based framework to directly learn the mapping from the corrupted image to the intact image and perform the pixel-level reconstruction. To perform semantic reasoning, our C2LGM introduces a Collaborative Contrastive Learning (C2L) mechanism that learns highlevel semantic consistency between inferred and known content. Specifically, C2L mechanism introduces the high-frequency edge maps to participate in the process of typical contrastive learning and enables the deep model to ensure the semantic reasonableness between high-frequency structures and pixel-level content by pushing the representations of inferred content and known content close and keeping unrelated semantic content away in the latent feature space. Moreover, C2LGM also directly absorbs the prior knowledge of structural information from the proposed structural spatial attention module, and leverages the texture distribution sampling to improve the quality of synthesized content. As a result, our C2LGM achieves a 0.42 dB improvement over competing methods in terms of the PSNR metric while coping with a 40∼ 50% corruption ratio in the Places2 dataset. Extensive experiments on three benchmark datasets, including Paris Street View, CelebA-HQ, and Places2, demonstrate the advantages of our proposed C2LGM over other state-of-the-art methods for image inpainting both qualitatively and quantitatively.INDEX TERMS Image inpainting, semantic reasoning, contrastive learning, generative model.

show abstract

“…The developed artificial intelligence algorithmic system consists of two networks. Convolutional neural network (CNN) reconstructs the low-resolution image in the frequency domain [12][13][14][15] . Then, the high-resolution image is predicted with the Generative Adversarial Network 16 (GAN) algorithm.…”

Section: Introductionmentioning

confidence: 99%

An AI-based algorithmic system that predicts missing A-scans in cross-sectional retinal images

Dinç,

Arli,

Tozburun

2023

Optical Coherence Imaging Techniques and Imaging in Scattering Media V

View full text Add to dashboard Cite

In this study, we present an artificial intelligence based algorithmic system that predicts missing A-scans of the edited OCT image by padding the A scan with zero. The developed artificial intelligence algorithmic system consists of two networks: convolutional neural network and generative adversarial network. The system theoretically suggests that skipping one-third of sequential A-scans and predicting the missing A-scan can increase the image acquisition rate by at least 33%. The structural similarity index measurement of the test data reaches an average of 82% between the ground truth images and the images predicted from the developed system. The mean squared error also is to 0.2%.

show abstract

Image inpainting using frequency-domain priors

Cited by 16 publications

References 37 publications

A deep learning image inpainting method based on stationary wavelet transform

A deep learning image inpainting method based on stationary wavelet transform

Collaborative Contrastive Learning-Based Generative Model for Image Inpainting

An AI-based algorithmic system that predicts missing A-scans in cross-sectional retinal images

Contact Info

Product

Resources

About