Structural similarity preserving GAN for infrared and visible image fusion

Abstract

Compared with a single image, in a complex environment, image fusion can utilize the complementary information provided by multiple sensors to significantly improve the image clarity and the information, more accurate, reliable, comprehensive access to target and scene information. It is widely used in military and civil fields, such as remote sensing, medicine, security and other fields. In this paper, we propose an end-to-end fusion framework based on structural similarity preserving GAN (SSP-GAN) to learn a mapping of the fusion tasks for visible and infrared images. Specifically, on the one hand, for making the fusion image natural and conforming to visual habits, structure similarity is introduced to guide the generator network produce abundant texture structure information. On the other hand, to fully take advantage of shallow detail information and deep semantic information for achieving feature reuse, we redesign the network architecture of multi-modal image fusion meticulously. Finally, a wide range of experiments on real infrared and visible TNO dataset and RoadScene dataset prove the superior performance of the proposed approach in terms of accuracy and visual. In particular, compared with the best results of other seven algorithms, our model has improved entropy, edge information transfer factor, multi-scale structural similarity and other evaluation metrics, respectively, by 3.05%, 2.4% and 0.7% on TNO dataset. And our model has also improved by 0.7%, 2.82% and 1.1% on RoadScene dataset.

Fusion of Infrared and Visible Images Based on Three-Scale Decomposition and ResNet Feature Transfer

“…Ma et al proposed the first image fusion method based on a generative adversarial network (GAN) [ 16 ], which transformed the fusion task into an adversarial learning process of infrared and visible image information retention, which opened up a new idea for the research of deep learning fusion methods. Zhang et al [ 17 ] proposed a GAN image fusion algorithm based on the preservation of structural similarity. The experiments show that this method has improved various indicators compared with the previous methods.…”