A New Deep Learning Based Multi-Spectral Image Fusion Method

Abstract

In this paper, we present a new effective infrared (IR) and visible (VIS) image fusion method by using a deep neural network. In our method, a Siamese convolutional neural network (CNN) is applied to automatically generate a weight map which represents the saliency of each pixel for a pair of source images. A CNN plays a role in automatic encoding an image into a feature domain for classification. By applying the proposed method, the key problems in image fusion, which are the activity level measurement and fusion rule design, can be figured out in one shot. The fusion is carried out through the multi-scale image decomposition based on wavelet transform, and the reconstruction result is more perceptual to a human visual system. In addition, the visual qualitative effectiveness of the proposed fusion method is evaluated by comparing pedestrian detection results with other methods, by using the YOLOv3 object detector using a public benchmark dataset. The experimental results show that our proposed method showed competitive results in terms of both quantitative assessment and visual quality.

Illumination-aware image fusion for around-the-clock human detection in adverse environments from Unmanned Aerial Vehicle

“…Image registration is another process to align and match two or more images from different cameras. A good optical and thermal image fusion method should be able to keep the thermal radiation information in thermal images and the texture detail information in optical images (Piao et al, 2019). The registration of optical and thermal images is a vital preliminary step for image fusion, object detection and tracking, and remote sensing to eliminate the offset between images (Yu et al, 2019;Ding et al, 2021;Dandrifosse et al, 2021).…”

A Generative Adversarial Network for Infrared and Visible Image Fusion Based on Semantic Segmentation

“…Recently, since the rise of deep learning, a large number of deep learning-based fusion methods have been proposed [ 26 , 27 ]. Li et al [ 28 ] decomposed infrared and visible images into base parts and detail content.…”

Multi-Source and Multi-Temporal Image Fusion on Hypercomplex Bases

“…Since then, it is used more and more in computer vision and image processing for classification and segmentation [32]. The approach is based on artificial neural networks like Convolutional Neural Networks (CNN) [57][58][59], Convolutional Sparse Coding [57,60], or Stacked Autoencoders [61]. The main advantage of this approach lies in its ability to outperform nearly every other machine learning algorithm.…”