Aggregated Contextual Transformations for High-Resolution Image Inpainting

Zeng, Yanhong; Fu, Jianlong; Chao, Hongyang; Guo, Baining

doi:10.1109/tvcg.2022.3156949

Cited by 143 publications

(108 citation statements)

References 59 publications

Supporting

Mentioning

108

Contrasting

Order By: Relevance

“…We have selected three image inpainting models, namely GMCNN [24], AOTGAN [25] and EdgeConnect [26], to explore the capability of inpainting polyps on a given clean colon image. These three models were chosen due to their popularity and novelty.…”

Section: Polyp Inpaintingmentioning

confidence: 99%

PolypConnect: Image inpainting for generating realistic gastrointestinal tract images with polyps

Fagereng¹,

Thambawita²,

Storås³

et al. 2022

Preprint

View full text Add to dashboard Cite

Early identification of a polyp in the lower gastrointestinal (GI) tract can lead to prevention of life-threatening colorectal cancer. Developing computer-aided diagnosis (CAD) systems to detect polyps can improve detection accuracy and efficiency and save the time of the domain experts called endoscopists. Lack of annotated data is a common challenge when building CAD systems. Generating synthetic medical data is an active research area to overcome the problem of having relatively few true positive cases in the medical domain. To be able to efficiently train machine learning (ML) models, which are the core of CAD systems, a considerable amount of data should be used. In this respect, we propose the PolypConnect pipeline, which can convert non-polyp images into polyp images to increase the size of training datasets for training. We present the whole pipeline with quantitative and qualitative evaluations involving endoscopists. The polyp segmentation model trained using synthetic data, and real data shows a 5.1% improvement of mean intersection over union (mIOU), compared to the model trained only using real data. The codes of all the experiments are available on GitHub to reproduce the results.

show abstract

Section: Polyp Inpaintingmentioning

confidence: 99%

PolypConnect: Image inpainting for generating realistic gastrointestinal tract images with polyps

Fagereng¹,

Thambawita²,

Storås³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…There are a few recent methods than build on STTN to generate higher resolution inpainted textures. These include introducing an Aggregated Contextual-Transformation GAN (AOT-GAN) Zeng et al (2021), combining 3D CNNs with a temporal shift and align module Zou et al (2021), and introducing a Deformable Alignment and Pyramid Context Completion Network with temporal attention Wu et al (2021). Addition-ally, more complex occlusions can be handled with a Decoupled Spatial-Temporal Transformer with a hierarchical encoder Liu et al (2021).…”

Section: Temporal Video Inpaintingmentioning

confidence: 99%

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence

Daher¹,

Vasconcelos²,

Stoyanov³

2022

Preprint

View full text Add to dashboard Cite

Video streams are utilised to guide minimally-invasive surgery and diagnostic procedures in a wide range of procedures, and many computer assisted techniques have been developed to automatically analyse them. These approaches can provide additional information to the surgeon such as lesion detection, instrument navigation, or anatomy 3D shape modeling. However, the necessary image features to recognise these patterns are not always reliably detected due to the presence of irregular light patterns such as specular highlight reflections. In this paper, we aim at removing specular highlights from endoscopic videos using machine learning. We propose using a temporal generative adversarial network (GAN) to inpaint the hidden anatomy under specularities, inferring its appearance spatially and from neighbouring frames where they are not present in the same location. This is achieved using in-vivo data of gastric endoscopy (Hyper-Kvasir) in a fully unsupervised manner that relies on automatic detection of specular highlights. System evaluations show significant improvements to traditional methods through direct comparison as well as other machine learning techniques through an ablation study that depicts the importance of the network's temporal and transfer learning components. The generalizability of our system to different surgical setups and procedures was also evaluated qualitatively on in-vivo data of gastric endoscopy and ex-vivo porcine data (SERV-CT, SCARED). We also assess the effect of our method in computer vision tasks that underpin 3D reconstruction and camera motion estimation, namely stereo disparity, optical flow, and sparse point feature matching. These are evaluated quantitatively and qualitatively and results show a positive effect of specular highlight inpainting on these tasks in a novel comprehensive analysis.

show abstract

“…[43] propose a mask awareness method using cascaded refinement network. Zeng et al utilize "AOT Block" and "SoftGAN" to enhance the generator and discriminator [38]. A new GAN called "Co-Modulated GAN" combining conditional GAN and modulated GAN is introduced in [41].…”

Section: Related Work 21 Image Inpaintingmentioning

confidence: 99%

“…Recent years, most state-of-the-art approaches are mainly based on convolutional neural networks or transformer. In the approaches of [22,35,38,40], they apply the convolutional neural networks for image inpainting, while other line of research [33,37] leverages the transformer in image inpainting at the low-resolution image space, and then introduces the GAN based networks for high quality image generation. Suvorov et al [31] utilize the Fast Fourier Convolution (FFC) instead of regular convolution to obtain features of global receptive fields in frequency domain.…”

Section: Introductionmentioning

confidence: 99%

GLaMa: Joint Spatial and Frequency Loss for General Image Inpainting

Lu¹,

Jiang²,

Huang³

et al. 2022

Preprint

View full text Add to dashboard Cite

The purpose of image inpainting is to recover scratches and damaged areas using context information from remaining parts. In recent years, thanks to the resurgence of convolutional neural networks (CNNs), image inpainting task has made great breakthroughs. However, most of the work consider insufficient types of mask, and their performance will drop dramatically when encountering unseen masks. To combat these challenges, we propose a simple yet general method to solve this problem based on the LaMa image inpainting framework [35], dubbed GLaMa. Our proposed GLaMa can better capture different types of missing information by using more types of masks. By incorporating more degraded images in the training phase, we can expect to enhance the robustness of the model with respect to various masks. In order to yield more reasonable results, we further introduce a frequency-based loss in addition to the traditional spatial reconstruction loss and adversarial loss. In particular, we introduce an effective reconstruction loss both in the spatial and frequency domain to reduce the chessboard effect and ripples in the reconstructed image. Extensive experiments demonstrate that our method can boost the performance over the original LaMa method for each type of mask on FFHQ [18], ImageNet [7], Places2 [42] and WikiArt [28] dataset. The proposed GLaMa was ranked first in terms of PSNR, LPIPS [39] and SSIM [34] in the NTIRE 2022 Image Inpainting Challenge Track 1 Unsupervised [27].

show abstract

Aggregated Contextual Transformations for High-Resolution Image Inpainting

Cited by 143 publications

References 59 publications

PolypConnect: Image inpainting for generating realistic gastrointestinal tract images with polyps

PolypConnect: Image inpainting for generating realistic gastrointestinal tract images with polyps

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence

GLaMa: Joint Spatial and Frequency Loss for General Image Inpainting

Contact Info

Product

Resources

About