TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network

Rao, Dongyu; Wu, Xiaojun; Xu, Tianyang

doi:10.48550/arxiv.2201.10147

Cited by 4 publications

(2 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, inspired by the work of Swin Transformer [45] , Ma et al [13] proposed Swin Fusion, a network architecture for multimodal fusion. In addition, Rao et al [46] proposed TGFuse, which embedded the Transformer in a GAN-based fusion network to achieve IVF. Furthermore, Qu et al [26] proposed TransMEF for multi-exposure image fusion, which combined CNN and Transformer to obtain powerful local modeling and global modeling capabilities.…”

Section: Transformer-based Fusion Methodsmentioning

confidence: 99%

DPACFuse: Dual-Branch Progressive Learning for Infrared and Visible Image Fusion with Complementary Self-Attention and Convolution

Zhu

Wang

et al. 2023

Sensors

View full text Add to dashboard Cite

Infrared and visible image fusion aims to generate a single fused image that not only contains rich texture details and salient objects, but also facilitates downstream tasks. However, existing works mainly focus on learning different modality-specific or shared features, and ignore the importance of modeling cross-modality features. To address these challenges, we propose Dual-branch Progressive learning for infrared and visible image fusion with a complementary self-Attention and Convolution (DPACFuse) network. On the one hand, we propose Cross-Modality Feature Extraction (CMEF) to enhance information interaction and the extraction of common features across modalities. In addition, we introduce a high-frequency gradient convolution operation to extract fine-grained information and suppress high-frequency information loss. On the other hand, to alleviate the CNN issues of insufficient global information extraction and computation overheads of self-attention, we introduce the ACmix, which can fully extract local and global information in the source image with a smaller computational overhead than pure convolution or pure self-attention. Extensive experiments demonstrated that the fused images generated by DPACFuse not only contain rich texture information, but can also effectively highlight salient objects. Additionally, our method achieved approximately 3% improvement over the state-of-the-art methods in MI, Qabf, SF, and AG evaluation indicators. More importantly, our fused images enhanced object detection and semantic segmentation by approximately 10%, compared to using infrared and visible images separately.

show abstract

Section: Transformer-based Fusion Methodsmentioning

confidence: 99%

DPACFuse: Dual-Branch Progressive Learning for Infrared and Visible Image Fusion with Complementary Self-Attention and Convolution

Zhu

Wang

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…DGLT-Fusion [40] decouples global-local information learning into Transformer and CNN modules, which enables the network to extract better global-local information. TGFuse [41] was proposed as a fusion algorithm that combines Transformer and GAN. The Transformer module is simply used to learn the global fusion relationship.…”

Section: Transformer For Image Fusionmentioning

confidence: 99%

SFPFusion: An Improved Vision Transformer Combining Super Feature Attention and Wavelet-Guided Pooling for Infrared and Visible Images Fusion

Li,

Xiao,

Cheng

et al. 2023

Sensors

View full text Add to dashboard Cite

The infrared and visible image fusion task aims to generate a single image that preserves complementary features and reduces redundant information from different modalities. Although convolutional neural networks (CNNs) can effectively extract local features and obtain better fusion performance, the size of the receptive field limits its feature extraction ability. Thus, the Transformer architecture has gradually become mainstream to extract global features. However, current Transformer-based fusion methods ignore the enhancement of details, which is important to image fusion tasks and other downstream vision tasks. To this end, a new super feature attention mechanism and the wavelet-guided pooling operation are applied to the fusion network to form a novel fusion network, termed SFPFusion. Specifically, super feature attention is able to establish long-range dependencies of images and to fully extract global features. The extracted global features are processed by wavelet-guided pooling to fully extract multi-scale base information and to enhance the detail features. With the powerful representation ability, only simple fusion strategies are utilized to achieve better fusion performance. The superiority of our method compared with other state-of-the-art methods is demonstrated in qualitative and quantitative experiments on multiple image fusion benchmarks.

show abstract

Mathematical modeling and simulation of multi-focus image fusion techniques using the effect of image enhancement criteria: a systematic review and performance evaluation

Choudhary

Sethi

2023

Artif Intell Rev

View full text Add to dashboard Cite

TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network

Cited by 4 publications

References 48 publications

DPACFuse: Dual-Branch Progressive Learning for Infrared and Visible Image Fusion with Complementary Self-Attention and Convolution

DPACFuse: Dual-Branch Progressive Learning for Infrared and Visible Image Fusion with Complementary Self-Attention and Convolution

SFPFusion: An Improved Vision Transformer Combining Super Feature Attention and Wavelet-Guided Pooling for Infrared and Visible Images Fusion

Mathematical modeling and simulation of multi-focus image fusion techniques using the effect of image enhancement criteria: a systematic review and performance evaluation

Contact Info

Product

Resources

About