Image Super-Resolution by Neural Texture Transfer

Abstract

Due to the significant information loss in low-resolution (LR) images, it has become extremely challenging to further advance the state-of-the-art of single image super-resolution (SISR). Reference-based super-resolution (RefSR), on the other hand, has proven to be promising in recovering high-resolution (HR) details when a reference (Ref) image with similar content as that of the LR input is given. However, the quality of RefSR can degrade severely when Ref is less similar. This paper aims to unleash the potential of RefSR by leveraging more texture details from Ref images with stronger robustness even when irrelevant Ref images are provided. Inspired by the recent work on image stylization, we formulate the RefSR problem as neural texture transfer. We design an end-to-end deep model which enriches HR details by adaptively transferring the texture from Ref images according to their textural similarity. Instead of matching content in the raw pixel space as done by previous methods, our key contribution is a multi-level matching conducted in the neural space. This matching scheme facilitates multi-scale neural transfer that allows the model to benefit more from those semantically related Ref patches, and gracefully degrade to SISR performance on the least relevant Ref inputs. We build a benchmark dataset for the general research of RefSR, which contains Ref images paired with LR inputs with varying levels of similarity. Both quantitative and qualitative evaluations demonstrate the superiority of our method over state-of-the-art 1 .

Axial Super-Resolution Study for Optical Coherence Tomography Images Via Deep Learning

“…With the development of deep learning methods, many kinds of networks have been applied to image recognition [16], [17], image denoising [18], [19], etc. Convolutional neural network (CNN) was firstly used for super-resolution (SR) image reconstruction in 2014 [20], and then different networks were proposed in reconstructing image details [21], especially for natural images and face images [22]- [24]. For medical imaging, such as X-CT [25], [26], MRI [27], [28], and ultrasound imaging [29], [30], some SR methods based on deep learning have also been proposed, which improved their spatial resolution effectively.…”

Super Resolution of B-Mode Ultrasound Images With Deep Learning

“…Gradient Magnitude Similarity Deviation (GMSD) metric is adapted to GMGAN [35] in order to produce HR images in line with the human vision system (HVS). The authors of SRNTT [36] focused on the information loss in LR images, and, proposed a reference-based SR approach for generating better texture details from reference images. A probabilistic generative framework, PGM, which offers the low computational cost and robustness to noise, is proposed in [37].…”

A multiresolution mixture generative adversarial network for video super-resolution

“…Although these methods have achieved excellent results according to PSNR and SSIM, the visual perception quality is still poor. Several models follow SRGANs [19] to combine the GAN and perceptual loss [21] to obtain ISR, which can generate improved results in visual quality [22][23][24][25].…”

“…The exchange unit of the MRMNet is shown in [24, 24, c] and the output dimension is [24,24,1024], where c denotes the number of channels of the input feature maps.…”