Cross-Modal Collaborative Representation Learning and a Large-Scale RGBT Benchmark for Crowd Counting

Abstract

Crowd counting is a fundamental yet challenging problem, which desires rich information to generate pixel-wise crowd density maps. However, most previous methods only utilized the limited information of RGB images and may fail to discover the potential pedestrians in unconstrained environments. In this work, we find that incorporating optical and thermal information can greatly help to recognize pedestrians. To promote future researches in this field, we introduce a large-scale RGBT Crowd Counting (RGBT-CC) benchmark, which contains 2,030 pairs of RGB-thermal images with 138,389 annotated people. Furthermore, to facilitate the multimodal crowd counting, we propose a cross-modal collaborative representation learning framework, which consists of multiple modalityspecific branches, a modality-shared branch, and an Information Aggregation-Distribution Module (IADM) to fully capture the complementary information of different modalities. Specifically, our IADM incorporates two collaborative information transfer components to dynamically enhance the modality-shared and modality-specific representations with a dual information propagation mechanism. Extensive experiments conducted on the RGBT-CC benchmark demonstrate the effectiveness of our framework for RGBT crowd counting. Moreover, the proposed approach is universal for multimodal crowd counting and is also capable to achieve superior performance on the ShanghaiTechRGBD [21] dataset.

Counting with Adaptive Auxiliary Learning

“…Recently, auxiliary task learning based counting methods [44], [45], [46], [47], [48], [49], [50], [51], [9], [52], [53] attracted researchers' attention because of its ability to capture extra granularity information and contextual dependencies for the density map regression. Most of the methods utilized the potential of a model itself with auxiliary tasks, such as object detection, crowd segmentation, density level classification, etc., to enhance the feature tuning for density map regression.…”

TAFNet: A Three-Stream Adaptive Fusion Network for RGB-T Crowd Counting

“…We compared the proposed method with several state-ofthe-art methods, including CSRNet [9], BL [7], CMCRL [3]. As shown in Table I, our method outperforms any other SOTA methods.…”

“…As shown in Fig. 3, after we get F i−T , F i−RGB and F i−C from i th stage, we apply pyramid pooling layer to extract contextual information CI i−T , CI i−RGB and CI i−C respectively, as [3] did. By using contextual information can we avoid excessive fusing of features and mitigate the misalignment in RGBT-CC dataset.…”

“…Our experiment was implemented with PyTorch [6]. We used BL [7] as the backbone and built our structure based on [3]. We used RGBT-CC [3] dataset which contains 1030, 200, 800 pairs of RGB-T images as training set, validation set and test set, respectively.…”

“…Therefore, how to fuse RGB and thermal information becomes the main problem. Liu et al [3] proposed a cross-modal collaborative representation learning framework and RGBT-CC dataset. Peng et al [17] proposed a novel end-to-end pipeline: MMCC, and DroneRGBT benchmark.…”