Translation, Scale and Rotation: Cross-Modal Alignment Meets RGB-Infrared Vehicle Detection

Yuan, Maoxun; Wang, Yinyan; Wei, Xingxing

doi:10.1007/978-3-031-20077-9_30

Cited by 22 publications

(4 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They introduced UA-CMDet, a method integrating visible, infrared, and bimodal fusion branches for detection. Yuan et al [38] developed TSFADet, aiming to mitigate the adverse effects of cross-modal misalignment by equalizing the discrepancies between two modal features. Zhang et al [39] proposed SuperYOLO, incorporating a cross-modal fusion module to extract supplementary information from the data.…”

Section: Multispectral Object Detection Algorithmmentioning

confidence: 99%

See 1 more Smart Citation

Multispectral Object Detection Based on Multilevel Feature Fusion and Dual Feature Modulation

Sun,

Yin,

Wang

et al. 2024

Electronics

View full text Add to dashboard Cite

Multispectral object detection is a crucial technology in remote sensing image processing, particularly in low-light environments. Most current methods extract features at a single scale, resulting in the fusion of invalid features and the failure to detect small objects. To address these issues, we propose a multispectral object detection network based on multilevel feature fusion and dual feature modulation (GMD-YOLO). Firstly, a novel dual-channel CSPDarknet53 network is used to extract deep features from visible-infrared images. This network incorporates a Ghost module, which generates additional feature maps through a series of linear operations, achieving a balance between accuracy and speed. Secondly, the multilevel feature fusion (MLF) module is designed to utilize cross-modal information through the construction of hierarchical residual connections. This approach strengthens the complementarity between different modalities, allowing the network to improve multiscale representation capabilities at a more refined granularity level. Finally, a dual feature modulation (DFM) decoupling head is introduced to enhance small object detection. This decoupled head effectively meets the distinct requirements of classification and localization tasks. GMD-YOLO is validated on three public visible-infrared datasets: DroneVehicle, KAIST, and LLVIP. DroneVehicle and LLVIP achieved mAP@0.5 of 78.0% and 98.0%, outperforming baseline methods by 3.6% and 4.4%, respectively. KAIST exhibited an MR of 7.73% with an FPS of 61.7. Experimental results demonstrated that our method surpasses existing advanced methods and exhibits strong robustness.

show abstract

Section: Multispectral Object Detection Algorithmmentioning

confidence: 99%

“…The evaluation results, as presented in Table 4, demonstrate significant improvements achieved by GMD-YOLO. [35] RGB + IR ResNet -73.9 UA-CMDet [37] RGB + IR YOLO -64.0 ECISNet [36] RGB + IR ResNet -76.0 TSFADet [38] RGB + IR ResNet -73.0 GMD-YOLO RGB + IR YOLO 80.3 78.0…”

Section: Comparison Experiments 441 Experiments On the Dronevehicle D...mentioning

confidence: 99%

Multispectral Object Detection Based on Multilevel Feature Fusion and Dual Feature Modulation

Sun,

Yin,

Wang

et al. 2024

Electronics

View full text Add to dashboard Cite

show abstract

“…UA-CMDet [23], RISNet [24], and ECISNet [25] optimize the cross-modal mutual information utilization to improve the detection performance. TSFADet [26] aligns cross-modal objects from translation, scaling, and rotation aspects through a network.…”

Section: Introductionmentioning

confidence: 99%

Improving YOLOv7-Tiny for Infrared and Visible Light Image Object Detection on Drones

Zhao

Lü

et al. 2023

Remote Sensing

View full text Add to dashboard Cite

To address the phenomenon of many small and hard-to-detect objects in drone images, this study proposes an improved algorithm based on the YOLOv7-tiny model. The proposed algorithm assigns anchor boxes according to the aspect ratio of ground truth boxes to provide prior information on object shape for the network and uses a hard sample mining loss function (HSM Loss) to guide the network to enhance learning from hard samples. This study finds that the aspect ratio difference of vehicle objects under drone perspective is more obvious than the scale difference, so the anchor boxes assigned by aspect ratio can provide more effective prior information for the network than those assigned by size. This study evaluates the algorithm on a drone image dataset (DroneVehicle) and compares it with other state-of-the-art algorithms. The experimental results show that the proposed algorithm achieves superior average precision values on both infrared and visible light images, while maintaining a light weight.

show abstract

“…Spatial misalignment: Since RGB and infrared sensors have different coordinate systems, fields of view, and sampling frequencies, pairs of RGB and infrared images usually are spatial misaligned [21], resulting in low-quality bounding boxes predicted by RGB and infrared fusion object detection, as shown in Figure 1a. Figure 1b indicates that the fusion of original RGB and infrared images via the image fusion algorithm [22] will take in a significantly misaligned ghost, which will also disturb localization.…”

Section: Introductionmentioning

confidence: 99%

Misaligned RGB-Infrared Object Detection via Adaptive Dual-Discrepancy Calibration

He,

Wu,

Ngan

et al. 2023

Remote Sensing

View full text Add to dashboard Cite

Object detection based on RGB and infrared images has emerged as a crucial research area in computer vision, and the synergy of RGB-Infrared ensures the robustness of object-detection algorithms under varying lighting conditions. However, the RGB-IR image pairs captured typically exhibit spatial misalignment due to sensor discrepancies, leading to compromised localization performance. Furthermore, since the inconsistent distribution of deep features from the two modalities, directly fusing multi-modal features will weaken the feature difference between the object and the background, therefore interfering with the RGB-Infrared object-detection performance. To address these issues, we propose an adaptive dual-discrepancy calibration network (ADCNet) for misaligned RGB-Infrared object detection, including spatial discrepancy and domain-discrepancy calibration. Specifically, the spatial discrepancy calibration module conducts an adaptive affine transformation to achieve spatial alignment of features. Then, the domain-discrepancy calibration module separately aligns object and background features from different modalities, making the distribution of the object and background of the fusion feature easier to distinguish, therefore enhancing the effectiveness of RGB-Infrared object detection. Our ADCNet outperforms the baseline by 3.3% and 2.5% in mAP50 on the FLIR and misaligned M3FD datasets, respectively. Experimental results demonstrate the superiorities of our proposed method over the state-of-the-art approaches.

show abstract

Translation, Scale and Rotation: Cross-Modal Alignment Meets RGB-Infrared Vehicle Detection

Cited by 22 publications

References 34 publications

Multispectral Object Detection Based on Multilevel Feature Fusion and Dual Feature Modulation

Multispectral Object Detection Based on Multilevel Feature Fusion and Dual Feature Modulation

Improving YOLOv7-Tiny for Infrared and Visible Light Image Object Detection on Drones

Misaligned RGB-Infrared Object Detection via Adaptive Dual-Discrepancy Calibration

Contact Info

Product

Resources

About