Semantic-Supervised Infrared and Visible Image Fusion Via a Dual-Discriminator Generative Adversarial Network

Zhou, Huabing; Wu, Wenzhuo; Zhang, Yanduo; Ma, Jiayi; Ling, Haibin

doi:10.1109/tmm.2021.3129609

Cited by 73 publications

(18 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Experimental Settings 1) Datasets: We utilize the color and infrared image pairs from the MSRS [19], RoadSence [2], and M3FD datasets [80] to evaluate the proposed framework. We also compare our method with six state-of-the-art algorithms: FusionGAN [9], SDDGAN [29], GANMcC [20], SDNet [27], U2Fusion [2], and TarDAL [80]. SDNet and U2Fusion are fusion approaches based on CNN architectures, while FusionGAN, SDDGAN, GANMcC and TarDAL are based on generative models and their variants.…”

Section: Methodsmentioning

confidence: 99%

“…The existing methods usually convert the visible images stored in three channels (i.e., RGB channels) from RGB space to YCbCr space, and use the Y channel for fusion [27], [2]. After the single-channel fused image is generated, it needs to be converted to a three-channel image through post-processing [28], [29]. Since not all channels are presented in the input data, it is hard to construct the multi-channel distribution and extract multi-channel complementary information, resulting in color distortion.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models

Yue¹,

Fang²,

Xia³

et al. 2023

Preprint

View full text Add to dashboard Cite

Color plays an important role in human visual perception, reflecting the spectrum of objects. However, the existing infrared and visible image fusion methods rarely explore how to handle multi-spectral/channel data directly and achieve high color fidelity. This paper addresses the above issue by proposing a novel method with diffusion models, termed as Dif-Fusion, to generate the distribution of the multi-channel input data, which increases the ability of multi-source information aggregation and the fidelity of colors. In specific, instead of converting multi-channel images into single-channel data in existing fusion methods, we create the multi-channel data distribution with a denoising network in a latent space with forward and reverse diffusion process. Then, we use the denoising network to extract the multi-channel diffusion features with both visible and infrared information. Finally, we feed the multi-channel diffusion features to the multi-channel fusion module to directly generate the three-channel fused image. To retain the texture and intensity information, we propose multi-channel gradient loss and intensity loss. Along with the current evaluation metrics for measuring texture and intensity fidelity, we introduce a new evaluation metric to quantify color fidelity. Extensive experiments indicate that our method is more effective than other state-of-the-art image fusion methods, especially in color fidelity. The source code will be publicly available at https://github.com/GeoVectorMatrix/Dif-Fusion.

show abstract

Section: Methodsmentioning

confidence: 99%

mentioning

confidence: 99%

Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models

Yue¹,

Fang²,

Xia³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…To alleviate this problem, they specifically designed two discriminators to realize fusion balance, and exploited DDcGAN [32] to implement multiresolution fusion tasks. In addition, Zhou et al [33] developed SDDGAN where an information quantity discrimination block was designed to supervise semantic information of source images under the framework of dual-discriminator generative adversarial network. Ma et al [34] translated image fusion into multi-classification constraints, namely GANMcC, which proposed two multi-classification discriminators to generate a more balanced result.…”

Section: B Gan-based Fusion Methodsmentioning

confidence: 99%

Infrared and Visible Image Fusion via Interactive Compensatory Attention Adversarial Learning

Wang¹,

Shao²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

The existing generative adversarial fusion methods generally concatenate source images and extract local features through convolution operation, without considering their global characteristics, which tends to produce an unbalanced result and is biased towards the infrared image or visible image. Toward this end, we propose a novel end-to-end mode based on generative adversarial training to achieve better fusion balance, termed as interactive compensatory attention fusion network (ICAFusion). In particular, in the generator, we construct a multi-level encoderdecoder network with a triple path, and adopt infrared and visible paths to provide additional intensity and gradient information. Moreover, we develop interactive and compensatory attention modules to communicate their pathwise information, and model their long-range dependencies to generate attention maps, which can more focus on infrared target perception and visible detail characterization, and further increase the representation power for feature extraction and feature reconstruction. In addition, dual discriminators are designed to identify the similar distribution between fused result and source images, and the generator is optimized to produce a more balanced result. Extensive experiments illustrate that our ICAFusion obtains superior fusion performance and better generalization ability, which precedes other advanced methods in the subjective visual description and objective metric evaluation. Our codes will be public at https://github.com/Zhishe-Wang/ICAFusion.

show abstract

“…Therefore, Li et al introduced a multiscale attention mechanism in the GAN-based fusion framework [ 29 ] to encourage the generators and discriminators to focus more on the most distinguishing regions. Moreover, Zhou et al [ 30 ] developed a dual-discriminator generative adversarial network (SDDGAN) where an information quantity discrimination (IQD) block was designed to guide the image fusion progress and supervise semantic information of source images in the fused image. Reference [ 31 ] designed a unified gradient and intensity-discriminator generative adversarial network for gradient and intensity retention in different image-fusion tasks.…”

Section: Related Workmentioning

confidence: 99%

Infrared and Visible Image Fusion via Attention-Based Adaptive Feature Fusion

Wang

Kong

et al. 2023

Entropy

View full text Add to dashboard Cite

Infrared and visible image fusion methods based on feature decomposition are able to generate good fused images. However, most of them employ manually designed simple feature fusion strategies in the reconstruction stage, such as addition or concatenation fusion strategies. These strategies do not pay attention to the relative importance between different features and thus may suffer from issues such as low-contrast, blurring results or information loss. To address this problem, we designed an adaptive fusion network to synthesize decoupled common structural features and distinct modal features under an attention-based adaptive fusion (AAF) strategy. The AAF module adaptively computes different weights assigned to different features according to their relative importance. Moreover, the structural features from different sources are also synthesized under the AAF strategy before reconstruction, to provide a more entire structure information. More important features are thus paid more attention to automatically and advantageous information contained in these features manifests itself more reasonably in the final fused images. Experiments on several datasets demonstrated an obvious improvement of image fusion quality using our method.

show abstract

Semantic-Supervised Infrared and Visible Image Fusion Via a Dual-Discriminator Generative Adversarial Network

Cited by 73 publications

References 53 publications

Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models

Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models

Infrared and Visible Image Fusion via Interactive Compensatory Attention Adversarial Learning

Infrared and Visible Image Fusion via Attention-Based Adaptive Feature Fusion

Contact Info

Product

Resources

About