Feature dynamic alignment and refinement for infrared–visible image fusion: Translation robust fusion

Li, Huafeng; Zhao, Jian; Li, Jinxing; Yu, Zhengtao; Lu, Guangming

doi:10.1016/j.inffus.2023.02.011

Cited by 25 publications

(9 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the field of computer vision, Visual Transformer (VT) have demonstrated significant capabilities across various tasks such as image Fusion [49,50]), pedestrian detection [51], particularly excelling in multispectral detection tasks [52][53][54][55] where they can focus on important features scattered across different spectral bands. Their self-attention mechanism's ability to model long-range dependencies and capture global context is especially valuable.…”

Section: Vision Transformermentioning

confidence: 99%

Cross-modality feature fusion for night pedestrian detection

Feng,

Luo,

et al. 2024

Front. Phys.

View full text Add to dashboard Cite

Night pedestrian detection with visible image only suffers from the dilemma of high miss rate due to poor illumination conditions. Cross-modality fusion can ameliorate this dilemma by providing complementary information to each other through infrared and visible images. In this paper, we propose a cross-modal fusion framework based on YOLOv5, which is aimed at addressing the challenges of night pedestrian detection under low-light conditions. The framework employs a dual-stream architecture that processes visible images and infrared images separately. Through the Cross-Modal Feature Rectification Module (CMFRM), visible and infrared features are finely tuned on a granular level, leveraging their spatial correlations to focus on complementary information and substantially reduce uncertainty and noise from different modalities. Additionally, we have introduced a two-stage Feature Fusion Module (FFM), with the first stage introducing a cross-attention mechanism for cross-modal global reasoning, and the second stage using a mixed channel embedding to produce enhanced feature outputs. Moreover, our method involves multi-dimensional interaction, not only correcting feature maps in terms of channel and spatial dimensions but also applying cross-attention at the sequence processing level, which is critical for the effective generalization of cross-modal feature combinations. In summary, our research significantly enhances the accuracy and robustness of nighttime pedestrian detection, offering new perspectives and technical pathways for visual information processing in low-light environments.

show abstract

Section: Vision Transformermentioning

confidence: 99%

Cross-modality feature fusion for night pedestrian detection

Feng,

Luo,

et al. 2024

Front. Phys.

View full text Add to dashboard Cite

show abstract

“…Currently, attention mechanisms have been extensively employed in many computer tasks, including semantic segmentation (Kirillov et al 2023), image fusion (Li et al 2023) and so on. For the field of correspondence pruning, the introduction of attention mechanisms is beneficial to focus on inlier information and suppress redundant information, but it still necessitates some appropriate modifications.…”

Section: Attention Mechanism In Correspondence Pruningmentioning

confidence: 99%

Graph Context Transformation Learning for Progressive Correspondence Pruning

Guo,

Xiao,

Wang

et al. 2024

AAAI

View full text Add to dashboard Cite

Most of existing correspondence pruning methods only concentrate on gathering the context information as much as possible while neglecting effective ways to utilize such information. In order to tackle this dilemma, in this paper we propose Graph Context Transformation Network (GCT-Net) enhancing context information to conduct consensus guidance for progressive correspondence pruning. Specifically, we design the Graph Context Enhance Transformer which first generates the graph network and then transforms it into multi-branch graph contexts. Moreover, it employs self-attention and cross-attention to magnify characteristics of each graph context for emphasizing the unique as well as shared essential information. To further apply the recalibrated graph contexts to the global domain, we propose the Graph Context Guidance Transformer. This module adopts a confident-based sampling strategy to temporarily screen high-confidence vertices for guiding accurate classification by searching global consensus between screened vertices and remaining ones. The extensive experimental results on outlier removal and relative pose estimation clearly demonstrate the superior performance of GCT-Net compared to state-of-the-art methods across outdoor and indoor datasets.

show abstract

“…With the development of sensor technology, different types of sensors can provide a variety of modal information for semantic segmentation tasks. The information interaction between RGB and other modalities can improve the performance of multimodal tasks [21,[41][42][43][44][45][46][47][48]. Specifically, Li et al [21,41,42], and Xiao et al [44] improved the quality of infrared and visible image fusion through cross-modal interaction between RGB image and infrared image.…”

Section: Cross-modal Interactionmentioning

confidence: 99%

MIPANet: optimizing RGB-D semantic segmentation through multi-modal interaction and pooling attention

Zhang,

Xie

2024

Front. Phys.

View full text Add to dashboard Cite

The semantic segmentation of RGB-D images involves understanding objects appearances and spatial relationships within a scene, which necessitates careful consideration of multiple factors. In indoor scenes, the presence of diverse and disorderly objects, coupled with illumination variations and the influence of adjacent objects, can easily result in misclassifications of pixels, consequently affecting the outcome of semantic segmentation. We propose a Multi-modal Interaction and Pooling Attention Network (MIPANet) in response to these challenges. This network is designed to exploit the interactive synergy between RGB and depth modalities, aiming to enhance the utilization of complementary information and improve segmentation accuracy. Specifically, we incorporate a Multi-modal Interaction Module (MIM) into the deepest layers of the network. This module is engineered to facilitate the fusion of RGB and depth information, allowing for mutual enhancement and correction. Moreover, we introduce a Pooling Attention Module (PAM) at various stages of the encoder to enhance the features extracted by the network. The outputs of the PAMs at different stages are selectively integrated into the decoder through a refinement module to improve semantic segmentation performance. Experimental results demonstrate that MIPANet outperforms existing methods on two indoor scene datasets, NYU-Depth V2 and SUN-RGBD, by optimizing the insufficient information interaction between different modalities in RGB-D semantic segmentation. The source codes are available at https://github.com/2295104718/MIPANet.

show abstract

Feature dynamic alignment and refinement for infrared–visible image fusion: Translation robust fusion

Cited by 25 publications

References 62 publications

Cross-modality feature fusion for night pedestrian detection

Cross-modality feature fusion for night pedestrian detection

Graph Context Transformation Learning for Progressive Correspondence Pruning

MIPANet: optimizing RGB-D semantic segmentation through multi-modal interaction and pooling attention

Contact Info

Product

Resources

About