A Deep Multitask Convolutional Neural Network for Remote Sensing Image Super-Resolution and Colorization

Feng, Jianan; Jiang, Qian; Tseng, Ching-Hsun; Jin, Xin; Liu, Ling; Zhou, Wei; Yao, Shaowen

doi:10.1109/tgrs.2022.3154435

Cited by 18 publications

(9 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Subsequently, Feng et. al [35] proved the method using the Multi-scale Residual Block to extract features and the Information Recovery Architecture (IRA) with multi-scale information transfer is proposed to generate the color images. The above multiscale coloring methods inevitably lead to the loss of microdetails in the process of using macro-context information to ensure spatial consistency [36]- [38].…”

Section: Automatic Coloring Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Remote Sensing Colorization Based on Bidirectional Macro-Micro Adaptive Enhancement Network

Wang

Yang

et al. 2022

IEEE Access

View full text Add to dashboard Cite

The demand for re-colorization of remote sensing images is urgent since image quality is extremely deteriorated by haze or other noises occurring in the atmospheric layer. The most challenging issue is to restore the color information with respect to preserving spatial consistency as well as to obtain object salience in context with extremely imbalanced space structure, where the former requires learning stable macroscopic semantics while the latter needs to recover microscopic pixels. In this paper, we propose a Bidirectional Macro-Micro Adaptive Enhancement (BMMAEnet) framework by adopting three modules, i.e., the Downward Micro Enhancement (DME) module, the Upward Adaptive Macro Enhancement(UAME) module, and Macro-Micro Balance (MMB) module. Firstly, the DME module is designed by adding micro details as well as suppressing macro context during the multi-branch downsampling process to supplement missing pixel details. Secondly, UAME is proposed by adaptive selecting proper level of features during multi-branch upsampling process to strengthen macro semantic constraints. In addition, MMB is designed by embedding attention-guided local details and global semantics into the decoding features to balance micro and macro information within each branch. Comprehensive comparison and ablation experiments are implemented and verify the proposed method performs overpass SOTA methods not only in pixel color value restoration performance but also in human perceptive understanding.

show abstract

Section: Automatic Coloring Methodsmentioning

confidence: 99%

“…g: Feng et al's Method (2022) [35] Use the Multi-scale Residual Block to extract features, and then the final coloring is achieved through the Information Recovery Architecture (IRA) with multi-scale information transfer.…”

Section: D: Amazon Mechanical Turk(amt) Perception Testmentioning

confidence: 99%

Remote Sensing Colorization Based on Bidirectional Macro-Micro Adaptive Enhancement Network

Wang

Yang

et al. 2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Given the potency of deep networks, several models incorporating residual learning [22,23] and multiscale architecture [24,25] have been introduced to address the challenges associated with training exceptionally deep networks. In the realm of RS, Feng et al [26] developed a lightweight CNN structure to enhance hierarchical feature learning and extract feature representations. More recently, with the advent of the Transformer architecture, several CNN-based models have integrated self-attention modules to facilitate global information extraction.…”

Section: Introductionmentioning

confidence: 99%

Denoising Diffusion Probabilistic Model with Adversarial Learning for Remote Sensing Super-Resolution

Sui,

Wu,

Pun

2024

Remote Sensing

View full text Add to dashboard Cite

Single Image Super-Resolution (SISR) for image enhancement enables the generation of high spatial resolution in Remote Sensing (RS) images without incurring additional costs. This approach offers a practical solution to obtain high-resolution RS images, addressing challenges posed by the expense of acquisition equipment and unpredictable weather conditions. To address the over-smoothing of the previous SISR models, the diffusion model has been incorporated into RS SISR to generate Super-Resolution (SR) images with enhanced textural details. In this paper, we propose a Diffusion model with Adversarial Learning Strategy (DiffALS) to refine the generative capability of the diffusion model. DiffALS integrates an additional Noise Discriminator (ND) into the training process, employing an adversarial learning strategy on the data distribution learning. This ND guides noise prediction by considering the general correspondence between the noisy image in each step, thereby enhancing the diversity of generated data and the detailed texture prediction of the diffusion model. Furthermore, considering that the diffusion model may exhibit suboptimal performance on traditional pixel-level metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), we showcase the effectiveness of DiffALS through downstream semantic segmentation applications. Extensive experiments demonstrate that the proposed model achieves remarkable accuracy and notable visual enhancements. Compared to other state-of-the-art methods, our model establishes an improvement of 189 for Fréchet Inception Distance (FID) and 0.002 for Learned Perceptual Image Patch Similarity (LPIPS) in a SR dataset, namely Alsat, and achieves improvements of 0.4%, 0.3%, and 0.2% for F1 score, MIoU, and Accuracy, respectively, in a segmentation dataset, namely Vaihingen.

show abstract

“…For individuals relying on such data, the precision of remote sensing information plays a crucial role in determining the quality of their work and everyday life. However, limited by the physical equipment, the onefold sensor cannot guarantee the high spectral and high spatial resolution of the captured images at the same time [1], [2]. Sensors usually obtain either a high-resolution panchromatic (PAN) image or a high-spectral MS image.…”

Section: Introductionmentioning

confidence: 99%

Generating Adversarial Remote Sensing Images via Pan-Sharpening Technique

Yuan

Wei

2021

Proceedings of the 1st International Workshop on Adversarial Learning for Multimedia

View full text Add to dashboard Cite

Pan-sharpening, as one of the most commonly used techniques in remote sensing systems, aims to inject spatial details from panchromatic images into multi-spectral images to obtain high-resolution MS images. Since deep learning has received widespread attention because of its powerful fitting ability and efficient feature extraction, a variety of pan-sharpening methods have been proposed to achieve remarkable performance. However, current pan-sharpening methods usually require the paired PAN and MS images as the input, which limits their usage in some scenarios. To address this issue, in this paper, we observe that the spatial details from PAN images are mainly high-frequency cues, i.e., the edges reflect the contour of input PAN images. This motivates us to develop a PAN-agnostic representation to store some base edges, so as to compose the contour for the corresponding PAN image via them. As a result, we can perform the pan-sharpening task with only the MS image when inference. To this end, a memory-based network is adapted to extract and memorize the spatial details during the training phase and is used to replace the process of obtaining spatial information from PAN images when inference, which is called Memory-based Spatial Details Network (MSDN). We finally integrate the proposed MSDN module into the existing DL-based pan-sharpening methods to achieve an end-to-end pan-sharpening network. With extensive experiments on the Gaofen1 and WorldView-4 satellites, we verify that our method constructs good spatial details without PAN images and achieves the best performance. The code is available at https://github.com/Zhao-Tian-yi/Learning-to-Pan-sharpeningwith-Memories-of-Spatial-Details.git.

show abstract

A Deep Multitask Convolutional Neural Network for Remote Sensing Image Super-Resolution and Colorization

Cited by 18 publications

References 32 publications

Remote Sensing Colorization Based on Bidirectional Macro-Micro Adaptive Enhancement Network

Remote Sensing Colorization Based on Bidirectional Macro-Micro Adaptive Enhancement Network

Denoising Diffusion Probabilistic Model with Adversarial Learning for Remote Sensing Super-Resolution

Generating Adversarial Remote Sensing Images via Pan-Sharpening Technique

Contact Info

Product

Resources

About