Remote Sensing Image Fusion Based on Two-Stream Fusion Network

Remote sensing image fusion (also known as pan-sharpening) aims at generating high resolution multi-spectral (MS) image from inputs of a high spatial resolution single band panchromatic (PAN) image and a low spatial resolution multi-spectral image. Inspired by the astounding achievements of convolutional neural networks (CNNs) in a variety of computer vision tasks, in this paper, we propose a two-stream fusion network (TFNet) to address the problem of pan-sharpening. Unlike previous CNN based methods that consider pan-sharpening as a super resolution problem and perform pan-sharpening in pixel level, the proposed TFNet aims to fuse PAN and MS images in feature level and reconstruct the pan-sharpened image from the fused features. The TFNet mainly consists of three parts. The first part is comprised of two networks extracting features from PAN and MS images, respectively. The subsequent network fuses them together to form compact features that represent both spatial and spectral information of PAN and MS images, simultaneously. Finally, the desired high spatial resolution MS image is recovered from the fused features through an image reconstruction network. Experiments on Quickbird and GaoFen-1 satellite images demonstrate that the proposed TFNet can fuse PAN and MS images, effectively, and produce pan-sharpened images competitive with even superior to state of the arts.

show abstract

“…This is an extension of our work [25]. Compare with it, this paper provides a more comprehensive and systematic report of our work.…”

Section: Introductionmentioning

confidence: 55%

Remote sensing image fusion based on two-stream fusion network

2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Furthermore, the images are usually recent and the process of creating reference data comparatively less challenging. In most of the existing works, both multispectral and panchromatic channels usually exist, and can be fed into the deep learning network by way of a pansharpened multispectral image, or separately feeding the channels and combining the feature maps at different stages of the network (Bergado et al, 2018;Gaetano et al, 2018;Kolokoussis et al, 2011;Liu et al, 2018;Tan et al, 2018). Historical photographs, however, have lower spatial and spectral resolution and lower quality because of degradation.…”

Section: Introductionmentioning

confidence: 99%

Fully convolutional networks for land cover classification from historical panchromatic aerial photographs

Mboga

Grippa

Georganos

et al. 2020

ISPRS Journal of Photogrammetry and Remote Sensing

View full text Add to dashboard Cite

Historical aerial photographs provide salient information on the historical state of the landscape. The exploitation of these archives is often limited by accessibility and the timeconsuming process of digitizing the analogue copies at a high resolution and processing them with a proper photogrammetric workflow. Furthermore, these data are characterised by limited spectral information since it occurs very often in a single band. Our work presents a first application of deep learning for the extraction of land cover from historical aerial panchromatic photographs of the African cities of Goma, Bukavu and Bujumbura. We evaluate the suitability of deep learning for land cover generation from a challenging dataset of photographs from the 1940s and 1950s that covers large geographical extents and is characterised by radiometric variations between dates and locations. A fully convolutional approach is investigated by considering two network architectures with different strategies of exploiting contextual information: one used atrous convolutional layers without downsampling, whereas the second network has both downsampling and learned upsampling convolutional layers (U-NET). The networks are trained to detect three main classes namely, buildings, high vegetation and a mixed class of bare land and low vegetation class. High overall accuracies of > 90% in Goma-Gisenyi and Bukavu, and >85 % in Bujumbura are obtained. This work provides a novel methodology that outperforms a baseline standard machine learning classifier for the exploitation of the vast archives of historical aerial photographs that can aid long-term environmental baseline studies. Future work will entail developing domain adaptation strategies in order to make the trained network robust for different image mosaics.

show abstract

“…Then, in [27], the authors proposed a two-stream fusion network (TFNet) which fused the PAN image and MSI in the feature level. Two CNNs are used to extract the features of PAN and LR-MSI separately and the extracted features are fused by concatenating in the fusion neural network.…”

Section: Introductionmentioning

confidence: 99%

A Two-Stream Multiscale Deep Learning Architecture for Pan-Sharpening

Jie

Cai

et al. 2020

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

Pan-sharpening, which fuses the high resolution panchromatic (PAN) image and the low resolution multispectral image (MSI), is a hot topic in remote sensing. Recently, deep learning technology has been successfully applied in pansharpening. However, the existing methods ignore that the MSI and PAN image are at different resolution and use the same networks to extract features of the two images. To address this problem, we propose a two-stream deep learning architecture, called coupled multi-scale convolutional neural network (CMC), for pan-sharpening. The proposed network has 3 components, feature extraction subnetworks, fusion layer, and super-resolution subnetwork. In the feature extraction subnetworks, two subnetworks are used to extract the features of the MSI and PAN image, separately. Different sizes of convolutional kernels are used in the first layers due to the different spatial resolutions. Thus, the source images are mapped to the similar scale. Then a multi-scale asymmetric convolution factorization is used to extract features at different scales. In the fusion layer, the two feature extraction subnetworks are coupled. Features at the same scale are firstly summed and then the features of all scales are concatenated as one feature map. At last, a super-resolution subnetwork is used to generate the high resolution MSI. Experimental results on both synthetic and real data sets demonstrate that the proposed method outperforms the other state-of-the-art pan-sharpening methods.

show abstract

Remote Sensing Image Fusion Based on Two-Stream Fusion Network

Cited by 42 publications

References 20 publications

Remote sensing image fusion based on two-stream fusion network

Remote sensing image fusion based on two-stream fusion network

Fully convolutional networks for land cover classification from historical panchromatic aerial photographs

A Two-Stream Multiscale Deep Learning Architecture for Pan-Sharpening

Contact Info

Product

Resources

About