Image fusion based on generative adversarial network consistent with perception

Fu, Yu; Wu, Xiaojun; Durrani, T.S.

doi:10.1016/j.inffus.2021.02.019

Cited by 85 publications

(26 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Methods based on GAN are also widely used in image generation tasks [33], [34]. There are already some image fusion methods based on GAN [20], [22]. Adversarial learning is an important part of our approach.…”

Section: B Generative Adversarial Networkmentioning

confidence: 99%

“…But this also makes it unable to effectively learn specific tasks. In order to obtain better performance for specific fusion tasks, the end-toend image fusion methods [20], [21], [22] are proposed to learn more targeted network parameters through a specific network structure and loss function. This method is dedicated to training fusion tasks, which can usually achieve better fusion results.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network

Rao¹,

Wu²,

Xu³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

The end-to-end image fusion framework has achieved promising performance, with dedicated convolutional networks aggregating the multi-modal local appearance. However, long-range dependencies are directly neglected in existing CNN fusion approaches, impeding balancing the entire image-level perception for complex scenario fusion. In this paper, therefore, we propose an infrared and visible image fusion algorithm based on a lightweight transformer module and adversarial learning. Inspired by the global interaction power, we use the transformer technique to learn the effective global fusion relations. In particular, shallow features extracted by CNN are interacted in the proposed transformer fusion module to refine the fusion relationship within the spatial scope and across channels simultaneously. Besides, adversarial learning is designed in the training process to improve the output discrimination via imposing competitive consistency from the inputs, reflecting the specific characteristics in infrared and visible images. The experimental performance demonstrates the effectiveness of the proposed modules, with superior improvement against the state-of-the-art, generalising a novel paradigm via transformer and adversarial learning in the fusion task.

show abstract

Section: B Generative Adversarial Networkmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network

Rao¹,

Wu²,

Xu³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Image fusion currently includes multiple sub-tasks, such as infrared and visible image fusion [6], multi-focus image fusion [7], multi-exposure image fusion [8], etc.. Fusion models for different tasks have a certain versatility as they share the same underlying function of integrating multiple images. Therefore, image fusion algorithms can be simply divided into two types: traditional methods [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21] and deep learning-based methods [22], [23], [24], [25], [26].…”

Section: Introductionmentioning

confidence: 99%

Unsupervised Image Fusion Method based on Feature Mutual Mapping

Rao¹,

Wu²,

Xu³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Deep learning-based image fusion approaches have obtained wide attention in recent years, achieving promising performance in terms of visual perception. However, the fusion module in the current deep learning-based methods suffers from two limitations, i.e., manually designed fusion function, and input-independent network learning. In this paper, we propose an unsupervised adaptive image fusion method to address the above issues. We propose a feature mutual mapping fusion module and dual-branch multi-scale autoencoder. More specifically, we construct a global map to measure the connections of pixels between the input source images. Besides, we design a dual-branch multi-scale network through sampling transformation to extract discriminative image features. We further enrich feature representations of different scales through feature aggregation in the decoding process. Finally, we propose a modified loss function to train the network with efficient convergence property. Through sufficient training on infrared and visible image data sets, our method also shows excellent generalized performance in multi-focus and medical image fusion. Our method achieves superior performance in both visual perception and objective evaluation. Experiments prove that the performance of our proposed method on a variety of image fusion tasks surpasses other state-of-the-art methods, proving the effectiveness and versatility of our approach.

show abstract

“…However, these methods are non-end-to-end model, and fusion strategy still need to be manually designed. To address this drawback, the generative adversarial network (GAN) based methods [19][20][21] were developed to transform image fusion into an adversarial game. Typically, Ma et al [19] exploited FusionGAN where the discriminator continuously Fig.…”

Section: Introductionmentioning

confidence: 99%

Infrared and Visible Image Fusion via Interactive Compensatory Attention Adversarial Learning

Wang¹,

Shao²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

The existing generative adversarial fusion methods generally concatenate source images and extract local features through convolution operation, without considering their global characteristics, which tends to produce an unbalanced result and is biased towards the infrared image or visible image. Toward this end, we propose a novel end-to-end mode based on generative adversarial training to achieve better fusion balance, termed as interactive compensatory attention fusion network (ICAFusion). In particular, in the generator, we construct a multi-level encoderdecoder network with a triple path, and adopt infrared and visible paths to provide additional intensity and gradient information. Moreover, we develop interactive and compensatory attention modules to communicate their pathwise information, and model their long-range dependencies to generate attention maps, which can more focus on infrared target perception and visible detail characterization, and further increase the representation power for feature extraction and feature reconstruction. In addition, dual discriminators are designed to identify the similar distribution between fused result and source images, and the generator is optimized to produce a more balanced result. Extensive experiments illustrate that our ICAFusion obtains superior fusion performance and better generalization ability, which precedes other advanced methods in the subjective visual description and objective metric evaluation. Our codes will be public at https://github.com/Zhishe-Wang/ICAFusion.

show abstract

Image fusion based on generative adversarial network consistent with perception

Cited by 85 publications

References 58 publications

TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network

TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network

Unsupervised Image Fusion Method based on Feature Mutual Mapping

Infrared and Visible Image Fusion via Interactive Compensatory Attention Adversarial Learning

Contact Info

Product

Resources

About