ECFuse: Edge-Consistent and Correlation-Driven Fusion Framework for Infrared and Visible Image Fusion

Abstract

Infrared and visible image fusion (IVIF) aims to render fused images that maintain the merits of both modalities. To tackle the challenge in fusing cross-modality information and avoiding texture loss in IVIF, we propose a novel edge-consistent and correlation-driven fusion framework (ECFuse). This framework leverages our proposed edge-consistency fusion module to maintain rich and coherent edges and textures, simultaneously introducing a correlation-driven deep learning network to fuse the cross-modality global features and modality-specific local features. Firstly, the framework employs a multi-scale transformation (MST) to decompose the source images into base and detail layers. Then, the edge-consistent fusion module fuses detail layers while maintaining the coherence of edges through consistency verification. A correlation-driven fusion network is proposed to fuse the base layers containing both modalities’ main features in the transformation domain. Finally, the final fused spatial image is reconstructed by inverse MST. We conducted experiments to compare our ECFuse with both conventional and deep leaning approaches on TNO, LLVIP and M3FD datasets. The qualitative and quantitative evaluation results demonstrate the effectiveness of our framework. We also show that ECFuse can boost the performance in downstream infrared–visible object detection in a unified benchmark.

Decomposed Multilateral Filtering for Accelerating Filtering with Multiple Guidance Images

“…Sensor fusion can combine beneficial information from different sensors to generate a richer single image. Image signal fusion approaches have various applications: RGB and infrared image fusion [ 1 , 2 , 3 ], RGB and multispectral image fusion [ 4 ], intercolor RGB signal fusion [ 5 , 6 ], RGB and depth fusion [ 7 , 8 ], RGB and light fusion [ 9 ], RGB and computed edge fusion [ 10 ], different focal image fusion [ 11 , 12 ], CT and MRI signal fusion for medical image processing [ 3 ], Retinex-based enhancement [ 13 ], SAR and multispectral image fusion [ 14 ], and general signal fusion [ 15 ].…”