TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network

Rao, Dongyu; Xu, Tianyang; Wu, Xiao-Jun

doi:10.1109/tip.2023.3273451

Cited by 67 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, Rao et al [ 31 ] proposed a visible and infrared fusion (VIF) method combining transformers with generative adversarial networks (GANs). In this method, the generator integrates both spatial and channel transformers to form a transformer fusion module.…”

Section: Related Workmentioning

confidence: 99%

TGLFusion: A Temperature-Guided Lightweight Fusion Method for Infrared and Visible Images

Yan,

Zhao,

Miao

et al. 2024

Sensors

View full text Add to dashboard Cite

The fusion of infrared and visible images is a well-researched task in computer vision. These fusion methods create fused images replacing the manual observation of single sensor image, often deployed on edge devices for real-time processing. However, there is an issue of information imbalance between infrared and visible images. Existing methods often fail to emphasize temperature and edge texture information, potentially leading to misinterpretations. Moreover, these methods are computationally complex, and challenging for edge device adaptation. This paper proposes a method that calculates the distribution proportion of infrared pixel values, allocating fusion weights to adaptively highlight key information. It introduces a weight allocation mechanism and MobileBlock with a multispectral information complementary module, innovations which strengthened the model’s fusion capabilities, made it more lightweight, and ensured information compensation. Training involves a temperature-color-perception loss function, enabling adaptive weight allocation based on image pair information. Experimental results show superiority over mainstream fusion methods, particularly in the electric power equipment scene and publicly available datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

TGLFusion: A Temperature-Guided Lightweight Fusion Method for Infrared and Visible Images

Yan,

Zhao,

Miao

et al. 2024

Sensors

View full text Add to dashboard Cite

show abstract

“…Attention mechanisms were originally used in natural language processing, and due to their ability to reduce computational errors and improve model operation capabilities with a small number of parameters added, they have been widely applied in various network structures. In recent years, due to the introduction of attention mechanisms in GAN [37][38][39], GAN-based image fusion algorithms have achieved better fusion effects. However, how to effectively integrate attention mechanisms into networks to achieve optimal results, as well as the structure of attention mechanisms, still need to be further explored.…”

Section: Attention Mechanismmentioning

confidence: 99%

NS-Fuse: Noise-Suppressed Fusion of Infrared and Visible Images via Improved Generative Adversarial Network

Cheng,

2024

Preprint

View full text Add to dashboard Cite

Aiµing at the traditional discriµinator types based on the generated counterµeasure network, the coµplex network structure and the noise probleµs in infrared equipµent of the existing iµage fusion µethods, a noise suppression iµage fusion µethod based on the iµproved generated counterµeasure network (NS-fuse) is proposed. By effectively coµbining the attention µechanisµ with the generator, this µethod strengthens the control of the generator on local features and global features. Two pyraµid feature µatching discriµinators are introduced to identify the fused iµage generated by the generator in infrared and visible diµensions. The new loss function is applied to the generation counterµeasure network, and the loss function is optiµized through the confrontation between the generator and the discriµinator, so as to iµprove the quality of the fused iµage. In addition, in order to coµbat the coµµon Gaussian noise in infrared iµages, the µethod also introduces new noise saµples as interference input to the generator to iµprove the de-noising ability of the generator for fused iµages. The µethod is coµpared with nine iµage fusion algorithµs on three public datasets. The results show that NS-fuse is better than the µost advanced µethod in qualitative analysis and quantitative analysis, and the optiµal structure of NS-fuse network is also obtained through experiµental exploration. The experiµental results show that NS-fuse network can effectively reµove noise while iµproving the details of the fused iµage, which proves the feasibility and effectiveness of the fused iµage in coµplex environµent, and has a good engineering application prospect.Index Terms—Image fusion, generative adversarial network (GAN), fused image denoising, pyramid feature matching discriminator (PFM), attention mechanism.

show abstract

“…DGLT-Fusion [ 40 ] decouples global–local information learning into Transformer and CNN modules, which enables the network to extract better global–local information. TGFuse [ 41 ] was proposed as a fusion algorithm that combines Transformer and GAN. The Transformer module is simply used to learn the global fusion relationship.…”

Section: Related Workmentioning

confidence: 99%

SFPFusion: An Improved Vision Transformer Combining Super Feature Attention and Wavelet-Guided Pooling for Infrared and Visible Images Fusion

Li,

Xiao,

Cheng

et al. 2023

Sensors

View full text Add to dashboard Cite

The infrared and visible image fusion task aims to generate a single image that preserves complementary features and reduces redundant information from different modalities. Although convolutional neural networks (CNNs) can effectively extract local features and obtain better fusion performance, the size of the receptive field limits its feature extraction ability. Thus, the Transformer architecture has gradually become mainstream to extract global features. However, current Transformer-based fusion methods ignore the enhancement of details, which is important to image fusion tasks and other downstream vision tasks. To this end, a new super feature attention mechanism and the wavelet-guided pooling operation are applied to the fusion network to form a novel fusion network, termed SFPFusion. Specifically, super feature attention is able to establish long-range dependencies of images and to fully extract global features. The extracted global features are processed by wavelet-guided pooling to fully extract multi-scale base information and to enhance the detail features. With the powerful representation ability, only simple fusion strategies are utilized to achieve better fusion performance. The superiority of our method compared with other state-of-the-art methods is demonstrated in qualitative and quantitative experiments on multiple image fusion benchmarks.

show abstract

TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network

Cited by 67 publications

References 0 publications

TGLFusion: A Temperature-Guided Lightweight Fusion Method for Infrared and Visible Images

TGLFusion: A Temperature-Guided Lightweight Fusion Method for Infrared and Visible Images

NS-Fuse: Noise-Suppressed Fusion of Infrared and Visible Images via Improved Generative Adversarial Network

SFPFusion: An Improved Vision Transformer Combining Super Feature Attention and Wavelet-Guided Pooling for Infrared and Visible Images Fusion

Contact Info

Product

Resources

About