TFIV: Multigrained Token Fusion for Infrared and Visible Image via Transformer

Abstract

Infrared and visible image fusion aims to extract complementary features to synthesize a single fused image. Many methods employ convolutional neural networks (CNNs) to extract local features due to its translation invariance and locality. However, CNNs fail to consider the image's non-local self-similarity (NLss), though it can expand the receptive field by pooling operations, it still inevitably leads to information loss. In addition, the transformer structure extracts long-range dependence by considering the correlativity among all image patches, leading to information redundancy of such transformer-based methods. However, graph representation is more flexible than grid (CNN) or sequence (transformer structure) representation to address irregular objects, and graph can also construct the relationships among the spatially repeatable details or texture with far-space distance. Therefore, to address the above issues, it is significant to convert images into the graph space and thus adopt graph convolutional networks (GCNs) to extract NLss. This is because the graph can provide a fine structure to aggregate features and propagate information across the nearest vertices without introducing redundant information. Concretely, we implement a cascaded NLss extraction pattern to extract NLss of intra-and inter-modal by exploring interactions of different image pixels in intra-and inter-image positional distance. We commence by preforming GCNs on each intra-modal to aggregate features and propagate information to extract independent intra-modal NLss. Then, GCNs are performed on the concatenate intramodal NLss features of infrared and visible images, which can explore the cross-domain NLss of inter-modal to reconstruct the fused image. We progressively expand the kennel sizes and dilations to increase the receptive field of GCNs, when we extract intra-and inter-modal NLss. Ablation studies and extensive experiments illustrates the effectiveness and superiority of the proposed method on three datasets.

A Method for Extracting Photovoltaic Panels from High-Resolution Optical Remote Sensing Images Guided by Prior Knowledge

“…With deep learning networks demonstrating competitiveness in object detection [7,8], image segmentation [9,10], and target extraction [11,12], deep learning-based methods have shown superior performance in the field of PV panel extraction. Deep learning-based methods can be divided into two main categories: methods based on classification and semantic segmentation networks.…”

An Infrared and Visible Image Fusion Method Guided by Saliency and Gradient Information

“…In recent years, With the successful development of deep learning (DL), many DL-based fusion methods are proposed to solve the issues of the traditional methods. Specifically, these DL-based methods can fuse the images through an end-to-end way, associated with convolutional neural networks (CNNs) [22], [23], generative adversarial networks (GANs) [24]- [28], the transformer [29]- [33], or other DL models [34], [35]. Thus, these recent DL-based method can generate satisfactory fused results, significantly improving the application performance.…”

A fast hyperspectral change detection algorithm for agricultural crops based on low-rank matrix and morphological feature extraction

“…Such methods enable modeling of spectral signals without assuming or estimating specific statistical distributions, which can be particularly useful for modeling change trends. Simultaneously, morphological feature extraction has been proven to be a powerful tool in fields such as crop classification and detection (Bosilj et al, 2018;Li et al, 2023). Expressing the topological structure and morphological attributes among crops enables the introduction of spatial information, forming a more universally applicable and effective method for detecting agricultural changes.…”