Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems

Zhou, Kailai; Chen, Linsen; Cao, Xun

doi:10.1007/978-3-030-58523-5_46

Cited by 148 publications

(102 citation statements)

References 38 publications

Supporting

Mentioning

102

Contrasting

Order By: Relevance

“…We evaluated the proposed fusion method on the KAIST testing dataset in the reasonable setting and in all settings in comparison with ACF+T+THOG [ 14 ], Halfway Fusion [ 22 ], Fusion RPN+BDT [ 30 ], IAF R-CNN [ 28 ], IATDNN+IASS [ 27 ], CIAN [ 37 ], MSDS-RCNN [ 17 ], ARCNN [ 38 ], MBNet [ 39 ], and FusionCSPNet [ 18 ]. Among these detection methods, FusionCSPNet and our method were one-stage methods, and the rest were two-stage methods.…”

Section: Methodsmentioning

confidence: 99%

Attention Fusion for One-Stage Multispectral Pedestrian Detection

Cao

Yang

Zhao

et al. 2021

Sensors

View full text Add to dashboard Cite

Multispectral pedestrian detection, which consists of a color stream and thermal stream, is essential under conditions of insufficient illumination because the fusion of the two streams can provide complementary information for detecting pedestrians based on deep convolutional neural networks (CNNs). In this paper, we introduced and adapted a simple and efficient one-stage YOLOv4 to replace the current state-of-the-art two-stage fast-RCNN for multispectral pedestrian detection and to directly predict bounding boxes with confidence scores. To further improve the detection performance, we analyzed the existing multispectral fusion methods and proposed a novel multispectral channel feature fusion (MCFF) module for integrating the features from the color and thermal streams according to the illumination conditions. Moreover, several fusion architectures, such as Early Fusion, Halfway Fusion, Late Fusion, and Direct Fusion, were carefully designed based on the MCFF to transfer the feature information from the bottom to the top at different stages. Finally, the experimental results on the KAIST and Utokyo pedestrian benchmarks showed that Halfway Fusion was used to obtain the best performance of all architectures and the MCFF could adapt fused features in the two modalities. The log-average miss rate (MR) for the two modalities with reasonable settings were 4.91% and 23.14%, respectively.

show abstract

Section: Methodsmentioning

confidence: 99%

Attention Fusion for One-Stage Multispectral Pedestrian Detection

Cao

Yang

Zhao

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Miss Rate (lower, better) All Day Night ACF [1] 47.32% 42.57% 56.17% Halfway Fusion [21] 25.75% 24.88% 26.59% Fusion RPN+BF [5] 18.29% 19.57% 16.27% IAF R-CNN [10] 15.73% 14.55% 18.26% IATDNN+IASS [9] 14.95% 14.67% 15.72% CIAN [7] 14.12% 14.77% 11.13% MSDS-RCNN [6] 11.34% 10.53% 12.94% AR-CNN [18] 9.34% 9.94% 8.38% MBNet [12] 8.13% 8.28% 7.86% Ours (full dataset) 8.86% 10.01% 6.77% Ours (10.26% of data) 9.32% 10.13% 7.70% [3] 39.7% 36.1% 36.8% FuseNet [14] 45.6% 41.0% 43.9% RTFNet [13] 53.2% 45.8% 54.8% Ours (full dataset) 53.6% 46.8% 53.3% Ours (17.99% of data) 51.0% 46.6% 48.9% Table 3. mIoU comparisons on TOKYO Dataset.…”

Section: Methodsmentioning

confidence: 99%

“…It turns out that the half-way feature fusion outperforms early or late fusion. Moreover, [7,8] apply attention mechanisms to learn an automatic re-weighting of visible and thermal features in the fusion module; [9,10] utilize illumination information as a guidance for the adaptive fusion of both features; [11,12] alleviate the inconsistency between visible and thermal features to facilitate the optimization of a dual-modality network.…”

Section: Related Workmentioning

confidence: 99%

Deep Active Learning from Multispectral Data Through Cross-Modality Prediction Inconsistency

Zhang

Fromont

Lefèvre

et al. 2021

2021 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Data from multiple sensors provide independent and complementary information, which may improve the robustness and reliability of scene analysis applications. While there exist many large-scale labelled benchmarks acquired by a single sensor, collecting labelled multi-sensor data is more expensive and time-consuming. In this work, we explore the construction of an accurate multispectral (here, visible & thermal cameras) scene analysis system with minimal annotation efforts via an active learning strategy based on the cross-modality prediction inconsistency. Experiments on multispectral datasets and vision tasks demonstrate the effectiveness of our method. In particular, with only 10% of labelled data on KAIST multispectral pedestrian detection dataset, we obtain comparable performance as other fully supervised State-of-the-Art methods.

show abstract

“…The research focus has then shifted to feature-level fusion: [18,13] studied the optimal fusion "timing" in the detection network and came to the same conclusion that halfway feature fusion produces better results; [14] introduced an auxiliary segmentation task on the basis of halfway feature fusion for further performance improvements. [29,27] applied attention mechanisms to adaptively weigh the visible and thermal features in the feature fusion stage; [26,32] alleviated the inconsistency between visible and thermal features to facilitate the optimization process of a dualmodality network. Apart from these studies on featurelevel fusion, multiple decision-level fusion methods were suggested: [6,15] used illumination information to guide the fusion of predictions (decisions) from visible/thermal images or from day/night sub-networks; [30] discussed a confidence-aware fusion mechanism, where the disagreement between visible and thermal predictions is used to reweigh visible contributions, which could also be regarded as a decision-level fusion approach.…”

Section: Multispectral Scene Analysismentioning

confidence: 99%

“…Multispectral information fusion methods can be categorized into: image-level fusion, feature-level fusion and decision-level fusion. Architectures that implement a feature-level fusion, usually within a two-stream network architecture (one dedicated to each source), have been proven to outperform the other strategies, and are currently the most studied in the literature [18,13,14,29,27,32,26,7,21,8]. However, the computational overhead provided by two-stream networks is huge, which is particularly undesirable for software deployment on embedded devices.…”

Section: Introductionmentioning

confidence: 99%

Low-cost Multispectral Scene Analysis with Modality Distillation

Zhang

Fromont

Lefèvre

et al. 2022

2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

Despite its robust performance under various illumination conditions, multispectral scene analysis has not been widely deployed due to two strong practical limitations: 1) thermal cameras, especially high-resolution ones are much more expensive than conventional visible cameras; 2) the most commonly adopted multispectral architectures, twostream neural networks, nearly double the inference time of a regular mono-spectral model which makes them impractical in embedded environments. In this work, we aim to tackle these two limitations by proposing a novel knowledge distillation framework named Modality Distillation (MD). The proposed framework distils the knowledge from a high thermal resolution two-stream network with featurelevel fusion to a low thermal resolution one-stream network with image-level fusion. We show on different multispectral scene analysis benchmarks that our method can effectively allow the use of low-resolution thermal sensors with more compact one-stream networks.

show abstract

Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems

Cited by 148 publications

References 38 publications

Attention Fusion for One-Stage Multispectral Pedestrian Detection

Attention Fusion for One-Stage Multispectral Pedestrian Detection

Deep Active Learning from Multispectral Data Through Cross-Modality Prediction Inconsistency

Low-cost Multispectral Scene Analysis with Modality Distillation

Contact Info

Product

Resources

About