Multi-Image Super-Resolution for Remote Sensing using Deep Recurrent Networks

Arefin, Rifat; Michalski, Vincent; St-Charles, Pierre-Luc; Kalaitzis, Alfredo; Kim, Sookyung; Kahou, Samira Ebrahimi; Bengio, Yoshua

doi:10.1109/cvprw50498.2020.00111

Cited by 30 publications

(17 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Registration is a crucial process of MISR which aligns LR images against sub-pixel shifts and rotations. Some methods concatenate the medians of images or the first image with each LR image and apply convolutional layers [5,15,16]. Others [6,8,17] exploit traditional methods such as masked FFT NCC [18] which register images in the Fourier domain using normalized cross-correlation, and PWC-net [19].…”

Section: Registrationmentioning

confidence: 99%

“…HighRes-net [5] recursively fuses two images using shared convolutional layers. DeepSUM [16] utilizes FusionNet composed of 3D convolutional layers and instance normalization [15] fuses images using recurrent neural network and global average pooling (GAP). However, neither convolutional nor recurrent models are adequate in learning temporal relations because they are dependent on the order of the input images.…”

Section: Fusionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-image Super-resolution via Quality Map Associated Temporal Attention Network

Lee¹

2022

Preprint

View full text Add to dashboard Cite

With the rising interest in deep learning-based methods in remote sensing, neural networks have made remarkable advancements in multi-image fusion and super-resolution. To fully exploit the advantages of multi-image super-resolution, temporal attention is crucial as it allows a model to focus on reliable features rather than noises. Despite the presence of quality maps (QMs) that indicate noises in images, most of the methods tested in the PROBA-V dataset have not been used QMs for temporal attention. We present a quality map associated temporal attention network (QA-Net), a novel method that incorporates QMs into both feature representation and fusion processes for the first time. Low-resolution features are temporally attended by QM features in repeated multi-head attention modules. The proposed method achieved state-of-the-art results in the PROBA-V dataset.

show abstract

Section: Registrationmentioning

confidence: 99%

Section: Fusionmentioning

confidence: 99%

Multi-image Super-resolution via Quality Map Associated Temporal Attention Network

Lee¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…It consists in combining the latent LR representations in a recursive manner to obtain the global representation which is upsampled to obtain the super-resolved image. In [22], the input LRs are co-registered in a similar way as in DeepSUM, and treated as a sequence which is processed with a recurrent neural network. This theoretically allows the network to be used with a variable number of LRs, however such experiments were not reported.…”

Section: Related Workmentioning

confidence: 99%

“…Overall, the existing MISR networks either learn to co-register the LRs [17,21,22], or they operate on already co-registered LRs [19]. Importantly, the co-registration process modifies the data which may lead to losing important information-the feature extraction layers in these architectures either "see" an individual LR image (before co-registration), or they process a stack of coregistered feature maps.…”

Section: Related Workmentioning

confidence: 99%

A Graph Neural Network For Multiple-Image Super-Resolution

Tarasiewicz

Nalepa

Kawulok

2021

2021 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Super-resolution consists in reconstructing a high-resolution image from single or multiple low-resolution observations. Deep learning has been reported extremely successful for single-image superresolution, but its applications to the multiple-image scenarios are limited due to the challenges that arise from feeding a network with a stack of images with sub-pixel translations. In this paper, we introduce Magnet-a new graph neural network that benefits from representing the input low-resolution images as a graph. This enables us to exploit the sub-pixel shifts among the input images while preserving the original low-resolution pixel values for feature extraction and information fusion. Despite a relatively simple architecture, Magnet outperforms the state-of-the-art methods for multiple-image superresolution, and due to the flexible graph representation, it allows for using a variable number of low-resolution images for reconstruction.

show abstract

“…Investigations of SR in remote sensing have extensively explored the use of Deep Neural Networks (DNNs). Both Single Image Super Resolution (SISR) [4]- [7] and Multiple Image Super Resolution (MISR) [8]- [11] architectures to super-resolve satellite images have surpassed the performance of traditional SR techniques. Research exploring the nuances of remote sensing training data to enhance the performance of these DNNs is less common.…”

Section: Introductionmentioning

confidence: 99%

Super-Resolving Beyond Satellite Hardware Using Realistically Degraded Images

White,

Codoreanu,

Zuleta

et al. 2021

Preprint

View full text Add to dashboard Cite

Modern deep Super-Resolution (SR) networks have established themselves as valuable techniques in image reconstruction and enhancement. However, these networks are normally trained and tested on benchmark image data that lacks the typical image degrading noise present in real images. In this paper, we test the feasibility of using deep SR in real remote sensing payloads by assessing SR performance in reconstructing realistically degraded satellite images. We demonstrate that a state-of-the-art SR technique called Enhanced Deep Super-Resolution Network (EDSR), without domain specific pretraining, can recover encoded pixel data on images with poor ground sampling distance, provided the ground resolved distance is sufficient. However, this recovery varies amongst selected geographical types. Our results indicate that custom training has potential to further improve reconstruction of overhead imagery, and that new satellite hardware should prioritise optical performance over minimising pixel size as deep SR can overcome a lack of the latter but not the former.

show abstract

Multi-Image Super-Resolution for Remote Sensing using Deep Recurrent Networks

Cited by 30 publications

References 29 publications

Multi-image Super-resolution via Quality Map Associated Temporal Attention Network

Multi-image Super-resolution via Quality Map Associated Temporal Attention Network

A Graph Neural Network For Multiple-Image Super-Resolution

Super-Resolving Beyond Satellite Hardware Using Realistically Degraded Images

Contact Info

Product

Resources

About