Cross-modality interactive attention network for multispectral pedestrian detection

Zhang, Lu; Li, Yong; Zhang, Shifeng; Yang, Xu; Qiao, Hong; Huang, Kaizhu; Hussain, Amir

doi:10.1016/j.inffus.2018.09.015

Cited by 161 publications

(93 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hwang et al [24] propose an extended ACF method, leveraging aligned color-thermal image pairs for around-the-clock pedestrian detection. With the recent development of deep learning, the CNN-based methods [52,48,6,55,19,32] significantly improve the multispectral pedestrian detection performances. Liu et al [34] adopt the Faster R-CNN architecture and analyze different fusion stages within the CNN.…”

Section: Related Workmentioning

confidence: 99%

“…Challenges A common assumption of multispectral pedestrian detection is that the color-thermal image pairs are geometrically aligned [24,52,34,28,32,18,55]. However, the modalities are just weakly aligned in practice, which means there is the position shift between modalities, making one object has different positions on different modalities, see Figure 1(a).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection

Zhang

Zhu²,

Chen

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

148

120

View full text Add to dashboard Cite

Multispectral pedestrian detection has shown great advantages under poor illumination conditions, since the thermal modality provides complementary information for the color image. However, real multispectral data suffers from the position shift problem, i.e. the color-thermal image pairs are not strictly aligned, making one object has different positions in different modalities. In deep learning based methods, this problem makes it difficult to fuse the feature maps from both modalities and puzzles the CNN training. In this paper, we propose a novel Aligned Region CNN (AR-CNN) to handle the weakly aligned multispectral data in an end-to-end way. Firstly, we design a Region Feature Alignment (RFA) module to capture the position shift and adaptively align the region features of the two modalities. Secondly, we present a new multimodal fusion method, which performs feature re-weighting to select more reliable features and suppress the useless ones. Besides, we propose a novel RoI jitter strategy to improve the robustness to unexpected shift patterns of different devices and system settings. Finally, since our method depends on a new kind of labelling: bounding boxes that match each modality, we manually relabel the KAIST dataset by locating bounding boxes in both modalities and building their relationships, providing a new KAIST-Paired Annotation. Extensive experimental validations on existing datasets are performed, demonstrating the effectiveness and robustness of the proposed method. Code and data are available at: https://github.com/luzhang16/AR-CNN .

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection

Zhang

Zhu²,

Chen

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

148

120

View full text Add to dashboard Cite

show abstract

“…Here, CIAN [53] runs much faster as compared to all other detectors including ours because the input size it adopts is much smaller. It's worth mentioning that the way that MSDS-RCNN uses box-level segmentation as supervision during the training phase is quite different from our method since we remodel the process as saliency detection and use the predicted saliency map as the spatial weights.…”

Section: Structurementioning

confidence: 99%

Attention Based Multi-Layer Fusion of Multispectral Images for Pedestrian Detection

2020

View full text Add to dashboard Cite

Multispectral images are increasingly used for pedestrian detection. Preliminary fusion strategies would fail to exploit informative features from cross-spectral images, or worse, may introduce additional interference. In this paper, we propose an attention based multi-layer fusion network in the triple-stream deep convolutional neural network architecture for multispectral pedestrian detection. The effectiveness of multi-layer fusion is examined and verified in this work. Furthermore, a channel-wise attention module (CAM) and a spatial-wise attention module (SAM) are developed and incorporated into the network aimng at more subtle adjustment to weights of multispectral features along both the channel and spatial dimensions respectively. Channel-wise attention is trained with self-supervision while spatialwise attention is trained with external supervision as we remodel its learning process as saliency detection. Both attention-based weighting mechanisms are evaluated separately and then sequentially. Experimental results on the KAIST dataset show that the proposed multi-layer cross-spectral fusion R-CNN (CS-RCNN), with spatial-wise weighting applied alone, achieves state-of-the-art performance on all-day detection while outperforming compared methods at nighttime. INDEX TERMS Convolutional neural networks, pedestrian detection, image fusion, deep learning

show abstract

“…For our daily essential need like safety, security and for other purposes, an accurate and fast pedestrian detection technique is very essential. Pedestrian detection task in both color image and videos [29][30][31] has achieved an outstanding accuracy in recent times. There are various machine learning methods like -Support Vector Machine [32], AdaBoost [33], Decision Tree [34], Neural Networks [46] are used for detecting the pedestrian in color images.…”

Section: Introductionmentioning

confidence: 99%

Pedestrian Detection in Thermal Images Using Deep Saliency Map and Instance Segmentation

Rahman¹,

Raihan²,

Islam³

2021

IJIGSP

View full text Add to dashboard Cite

Pedestrian detection is an established instance of computer vision task. Pedestrian detection from the color images has achieved robust performance but in the night time or in bad light conditions it has low detection accuracy. Thermal images are used for detecting people at night time, foggy weather or in bad lighting situations when color images have a lower vision. But in the daytime where the surroundings are warm or warmer than pedestrians then the thermal image has lower accuracy. Hence thermal and color image pair can be a solution but it is expensive to capture color-thermal pair and misaligned imagery can cause low detection accuracy. We proposed a network that achieved better accuracy by extending the prior works which introduced the use of the saliency map in pedestrian detection tasks from the thermal images into instance-level segmentation. We worked on a subdivision of KAIST Multispectral Pedestrian Detection Dataset [8] which has pixel-level annotations. We have trained Mask-RCNN for pedestrian detection task and report the added effect of saliency maps generated using PiCA-Net. We have achieved an accuracy of 88.14% over day and 91.84% over night images.. So, our model has reduced the miss rate by 24.1% and 23% over the existing state-of-the-art method in day and night images.

show abstract

Cross-modality interactive attention network for multispectral pedestrian detection

Cited by 161 publications

References 17 publications

Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection

Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection

Attention Based Multi-Layer Fusion of Multispectral Images for Pedestrian Detection

Pedestrian Detection in Thermal Images Using Deep Saliency Map and Instance Segmentation

Contact Info

Product

Resources

About