Low-cost Multispectral Scene Analysis with Modality Distillation

Abstract

Despite its robust performance under various illumination conditions, multispectral scene analysis has not been widely deployed due to two strong practical limitations: 1) thermal cameras, especially high-resolution ones are much more expensive than conventional visible cameras; 2) the most commonly adopted multispectral architectures, twostream neural networks, nearly double the inference time of a regular mono-spectral model which makes them impractical in embedded environments. In this work, we aim to tackle these two limitations by proposing a novel knowledge distillation framework named Modality Distillation (MD). The proposed framework distils the knowledge from a high thermal resolution two-stream network with featurelevel fusion to a low thermal resolution one-stream network with image-level fusion. We show on different multispectral scene analysis benchmarks that our method can effectively allow the use of low-resolution thermal sensors with more compact one-stream networks.

Efficient RGB-T Tracking via Cross-Modality Distillation

“…In addition, KD is also employed for some multi-modal tasks, such as RGB-D salient object detection [19,20] and RGB-T pedestrian detection [15,29]. Specifically, in [20,29], the single-stream feature extractors and the early fusion strategies were employed in their student models.…”

“…In addition, KD is also employed for some multi-modal tasks, such as RGB-D salient object detection [19,20] and RGB-T pedestrian detection [15,29]. Specifically, in [20,29], the single-stream feature extractors and the early fusion strategies were employed in their student models. However, both of them only simply employ the distillation loss functions to improve the performance of student models by using fused features or label knowledges of teacher models, and pay less attention to the huge differences between the teacher model and the student model in the unimodal feature extraction stage as well as in the multi-modal feature fusion stage.…”

“…Teacher-Student knowledge distillation experiments. Table 4 shows the performance of using some other knowledge distillation methods in the feature extraction and feature fusion stages for comparisons, including KD [8], Fit-Nets [21], ReviewKD [3] and MD [29]. It is observed from Table 4 that the proposed distillation strategy performs the best.…”

“…compact model the student model. Although some works [15,19,20,29] have made considerable progress on knowledge distillation in multi-modal tasks, they fail to conduct a deep investigation on the huge feature differences between teacher and student in the unimodal feature extraction stage as well as in the multi-modal feature fusion stage, thereby resulting in suboptimal efficiency of the knowledge transformation. For that, a novel teacher-student knowledge distillation training framework, named Cross-Modality Distillation (CMD), is proposed to elaborately guide efficient imitation from three stages: unimodal feature extraction, multi-modal feature fusion and target estimate estimation, as shown in Fig.…”

TF-YOLO: A Transformer–Fusion-Based YOLO Detector for Multimodal Pedestrian Detection in Autonomous Driving Scenes

“…Therefore, this work focuses on improving the effectiveness of pedestrian detectors in low-light conditions. Combining multispectral images (visible and infrared images) has proven useful for robust pedestrian detection in ADAS applications [6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21]. The signals of visible and infrared images originate from different modes and can provide scene information from different aspects.…”