Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections

Mao, Xu; Shen, Chunhua; Yang, Yongchun

doi:10.48550/arxiv.1603.09056

Cited by 104 publications

(70 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The motion compensation module of VESPCN [54] was employed in [57], where sub-pixel motion compensation layer was introduced to perform simultaneous motion compensation and resolution enhancement. After the motion compensation, the frames are super-resolved using an encoder-decoder framework with skip connections [58] and ConvLSTM [59] module for faster convergence as well as to take care of the sequential nature of video. Previous end-to-end CNN based video SR methods have focused on explicit motion estimation and compensation to better re-construct HR frames.…”

Section: Deep Learning Based Approachesmentioning

confidence: 99%

Deep Networks for Image and Video Super-Resolution

Purohit¹,

Mandal²,

Rajagopalan³

2022

Preprint

View full text Add to dashboard Cite

Efficiency of gradient propagation in intermediate layers of convolutional neural networks is of key importance for superresolution task. To this end, we propose a deep architecture for single image super-resolution (SISR), which is built using efficient convolutional units we refer to as mixed-dense connection blocks (MDCB). The design of MDCB combines the strengths of both residual and dense connection strategies, while overcoming their limitations. To enable super-resolution for multiple factors, we propose a scale-recurrent framework which reutilizes the filters learnt for lower scale factors recursively for higher factors. This leads to improved performance and promotes parametric efficiency for higher factors. We train two versions of our network to enhance complementary image qualities using different loss configurations. We further employ our network for video super-resolution task, where our network learns to aggregate information from multiple frames and maintain spatio-temporal consistency. The proposed networks lead to qualitative and quantitative improvements over state-of-the-art techniques on image and video super-resolution benchmarks. IntroductionSingle image super-resolution (SISR) aims to estimate a high-resolution (HR) image from a low-resolution (LR) input image, and is an ill-posed problem. Due to its diverse applicability starting from surveillance to medical diagnosis, and from remote sensing to HDTV, the SISR problem has gathered substantial attention from computer vision and image processing community. The ill-posed nature of the problem is generally addressed by learning a LR-HR mapping function in a constrained environment using example HR-LR patch pairs.One way is to learn a mapping function that linearly correlates the HR-LR patch pairs. Such linear functions can be easily learned with few example images as has been practiced by some SR approaches [1,2,3]. But, linear mapping between such patch pairs may not be representative enough to learn different complex structures present in the image. The mapping function would benefit from learning non-linear relationships between HR-LR patch pairs. Recent convolutional neural network (CNN) based models are quite efficient for such a purpose, and can be useful in extracting relevant features by making deeper models. However, deeper models often face vanishing/exploding gradient issues, which can be partially mitigated by using residual mapping [4,5]. Deep residual models has been employed for higher level vision tasks, where batch normalization is generally used for a useful class-specific normalized representation. However, such representation is not much useful in low-level vision task such as SR [6]. Most deep CNN based SR models do not make full use of the hierarchical features from the original LR images. Thus, the scope of improvement in performance is there in effective employment of the hierarchical features from all the convolutional layers, as has been employed by a residual dense network [7] using a sequence of residual dense block...

show abstract

Section: Deep Learning Based Approachesmentioning

confidence: 99%

Deep Networks for Image and Video Super-Resolution

Purohit¹,

Mandal²,

Rajagopalan³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, with the advent of deep neural networks, supervised learning-based image denoisers [27,46,47] show promising performance on a set of clean and noisy image pairs. However, it is challenging to construct clean and noisy image pairs in a real-world scenario.…”

Section: Image Denoisingmentioning

confidence: 99%

“…DIP [35], N2N [25], N2V [24], and LIR [12], and supervised methods, i.e. DnCNN [46], FFDNet [47], and RedNet-30 [27], to compare the performance. Traditional Low-Pass Filtering (LPF) and BM3D [10] are also evaluated.…”

Section: Synthetic Noise Removalmentioning

confidence: 99%

See 1 more Smart Citation

Unsupervised Image Denoising with Frequency Domain Knowledge

Kim¹,

Jang²,

Lee³

et al. 2021

Preprint

View full text Add to dashboard Cite

Supervised learning-based methods yield robust denoising results, yet they are inherently limited by the need for large-scale clean/noisy paired datasets. The use of unsupervised denoisers, on the other hand, necessitates a more detailed understanding of the underlying image statistics. In particular, it is well known that apparent differences between clean and noisy images are most prominent on high-frequency bands, justifying the use of low-pass filters as part of conventional image preprocessing steps. However, most learning-based denoising methods utilize only one-sided information from the spatial domain without considering frequency domain information. To address this limitation, in this study we propose a frequency-sensitive unsupervised denoising method. To this end, a generative adversarial network (GAN) is used as a base structure. Subsequently, we include spectral discriminator and frequency reconstruction loss to transfer frequency knowledge into the generator. Results using natural and synthetic datasets indicate that our unsupervised learning method augmented with frequency information achieves stateof-the-art denoising performance, suggesting that frequency domain information could be a viable factor in improving the overall performance of unsupervised learning-based methods.

show abstract

“…Now, in a DNN, there may exist thousands or even millions of these latent features, rendering it virtually impossible to understand how the input is mapped to output. To further complicate matters, many networks have skip connections 2 , recurrent blocks 3 and convolutional operations 4 ) that further muddy the waters.…”

Section: Introductionmentioning

confidence: 99%

Brittle AI, Causal Confusion, and Bad Mental Models: Challenges and Successes in the XAI Program

Druce¹,

Niehaus²,

Moody³

et al. 2021

Preprint

View full text Add to dashboard Cite

The advances in artificial intelligence enabled by deep learning architectures are undeniable. In several cases, deep neural network driven models have surpassed human level performance in benchmark autonomy tasks. The underlying policies for these agents, however, are not easily interpretable. In fact, given their underlying deep models, it is impossible to directly understand the mapping from observations to actions for any reasonably complex agent. Producing this supporting technology to "open the black box" of these AI systems, while not sacrificing performance, was the fundamental goal of the DARPA XAI program. In our journey through this program, we have several "big picture" takeaways: 1) Explanations need to be highly tailored to their scenario; 2) many seemingly high performing RL agents are extremely brittle and are not amendable to explanation; 3) causal models allow for rich explanations, but how to present them isn't always straightforward; and 4) human subjects conjure fantastically wrong mental models for AIs, and these models are often hard to break. This paper discusses the origins of these takeaways, provides amplifying information, and suggestions for future work.

show abstract

Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections

Cited by 104 publications

References 30 publications

Deep Networks for Image and Video Super-Resolution

Deep Networks for Image and Video Super-Resolution

Unsupervised Image Denoising with Frequency Domain Knowledge

Brittle AI, Causal Confusion, and Bad Mental Models: Challenges and Successes in the XAI Program

Contact Info

Product

Resources

About