Learning Cross-Modal Deep Representations for Robust Pedestrian Detection

Xu, Dan; Ouyang, Wanli; Ricci, Elisa; Wang, Xiaogang; Sebe, Nicu

doi:10.1109/cvpr.2017.451

Cited by 200 publications

(106 citation statements)

References 43 publications

Supporting

Mentioning

106

Contrasting

Order By: Relevance

“…Among the sensors, thermal camera is widely used in face recognition [3,44,27], human tracking [30,46] and action recognition [59,15] for its biometric robustness. Motivated by this, multispectral pedestrian detection [24,52,17,39] has attracted massive attention and provides new opportunities for around-theclock applications, mainly due to its superiority of complementary nature between color and thermal modalities.…”

Section: Introductionmentioning

confidence: 99%

Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection

Zhang

Zhu²,

Chen

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

148

120

View full text Add to dashboard Cite

Multispectral pedestrian detection has shown great advantages under poor illumination conditions, since the thermal modality provides complementary information for the color image. However, real multispectral data suffers from the position shift problem, i.e. the color-thermal image pairs are not strictly aligned, making one object has different positions in different modalities. In deep learning based methods, this problem makes it difficult to fuse the feature maps from both modalities and puzzles the CNN training. In this paper, we propose a novel Aligned Region CNN (AR-CNN) to handle the weakly aligned multispectral data in an end-to-end way. Firstly, we design a Region Feature Alignment (RFA) module to capture the position shift and adaptively align the region features of the two modalities. Secondly, we present a new multimodal fusion method, which performs feature re-weighting to select more reliable features and suppress the useless ones. Besides, we propose a novel RoI jitter strategy to improve the robustness to unexpected shift patterns of different devices and system settings. Finally, since our method depends on a new kind of labelling: bounding boxes that match each modality, we manually relabel the KAIST dataset by locating bounding boxes in both modalities and building their relationships, providing a new KAIST-Paired Annotation. Extensive experimental validations on existing datasets are performed, demonstrating the effectiveness and robustness of the proposed method. Code and data are available at: https://github.com/luzhang16/AR-CNN .

show abstract

Section: Introductionmentioning

confidence: 99%

Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection

Zhang

Zhu²,

Chen

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

148

120

View full text Add to dashboard Cite

show abstract

“…Deep learning based methods for pedestrian detection [8,9,10,205,206,207,208,209,210,211] For pedestrian detection, one of the most significant challenges is to handle occlusion [216,217,218,219,220,221,222,223,224,225,214,226]. A straightforward method is to use part-based models which learn a series of part detectors and integrate the results of part detectors to locate and classify objects.…”

Section: Pedestrian Detectionmentioning

confidence: 99%

Recent advances in deep learning for object detection

2020

View full text Add to dashboard Cite

Object detection is a fundamental visual recognition problem in computer vision and has been widely studied in the past decades.Visual object detection aims to find objects of certain target classes with precise localization in a given image and assign each object instance a corresponding class label. Due to the tremendous successes of deep learning based image classification, object detection techniques using deep learning have been actively studied in recent years. In this paper, we give a comprehensive survey of recent advances in visual object detection with deep learning. By reviewing a large body of recent related work in literature, we systematically analyze the existing object detection frameworks and organize the survey into three major parts: (i) detection components, (ii) learning strategies, and (iii) applications & benchmarks. In the survey, we cover a variety of factors affecting the detection performance in detail, such as detector architectures, feature learning, proposal generation, sampling strategies, etc.Finally, we discuss several future directions to facilitate and spur future research for visual object detection with deep learning.

show abstract

“…The problem of RGBT tracking could be considered as an extension of visual tracking, and its goal is to estimate target states using the complementary advantages of visible spectrum (called RGB in this paper) and thermal infrared information given the initial state in the first pair of frame. It has been receiving much more attention re-cently and becoming more and more popular partly due to the following reasons: i) RGB and thermal data have strong complementary advantages and thus could overcome imaging limitations of individual source [19,31,32,22]. ii) Thermal infrared cameras are economically available in recent years [6], making RGBT data easier to access in various applications, such as object segmentation [19], person Re-ID [31] and pedestrian detection [10,32].…”

Section: Introductionmentioning

confidence: 99%

“…It has been receiving much more attention re-cently and becoming more and more popular partly due to the following reasons: i) RGB and thermal data have strong complementary advantages and thus could overcome imaging limitations of individual source [19,31,32,22]. ii) Thermal infrared cameras are economically available in recent years [6], making RGBT data easier to access in various applications, such as object segmentation [19], person Re-ID [31] and pedestrian detection [10,32]. iii) Recent RGBT tracking benchmark datasets [17,22] provide a flexible evaluation platform of various RGBT trackers.…”

Section: Introductionmentioning

confidence: 99%

Multi-Adapter RGBT Tracking

Zheng

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

143

116

View full text Add to dashboard Cite

The task of RGBT tracking aims to take the complementary advantages from visible spectrum and thermal infrared data to achieve robust visual tracking, and receives more and more attention in recent years. Existing works focus on modality-specific information integration by introducing modality weights to achieve adaptive fusion or learning robust feature representations of different modalities. Although these methods could effectively deploy the modality-specific properties, they ignore the potential values of modality-shared cues as well as instance-aware information, which are crucial for effective fusion of different modalities in RGBT tracking. In this paper, we propose a novel Multi-Adapter convolutional Network (MANet) to jointly perform modality-shared, modality-specific and instance-aware feature learning in an end-to-end trained deep framework for RGBT tracking. We design three kinds of adapters within our network. In a specific, the generality adapter is to extract shared object representations, the modality adapter aims at encoding modality-specific information to deploy their complementary advantages, and the instance adapter is to model the appearance properties and temporal variations of a certain object. Moreover, to reduce computational complexity for real-time demand of visual tracking, we design a parallel structure of generic adapter and modality adapter. Extensive experiments on two RGBT tracking benchmark datasets demonstrate the outstanding performance of the proposed tracker against other state-ofthe-art RGB and RGBT tracking algorithms.

show abstract

Learning Cross-Modal Deep Representations for Robust Pedestrian Detection

Cited by 200 publications

References 43 publications

Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection

Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection

Recent advances in deep learning for object detection

Multi-Adapter RGBT Tracking

Contact Info

Product

Resources

About