Object Tracking in RGB-T Videos Using Modal-Aware Attention Network and Competitive Learning

Abstract

Object tracking in RGB-thermal (RGB-T) videos is increasingly used in many fields due to the all-weather and all-day working capability of the dual-modality imaging system, as well as the rapid development of low-cost and miniaturized infrared camera technology. However, it is still very challenging to effectively fuse dual-modality information to build a robust RGB-T tracker. In this paper, an RGB-T object tracking algorithm based on a modal-aware attention network and competitive learning (MaCNet) is proposed, which includes a feature extraction network, modal-aware attention network, and classification network. The feature extraction network adopts the form of a two-stream network to extract features from each modality image. The modal-aware attention network integrates the original data, establishes an attention model that characterizes the importance of different feature layers, and then guides the feature fusion to enhance the information interaction between modalities. The classification network constructs a modality-egoistic loss function through three parallel binary classifiers acting on the RGB branch, the thermal infrared branch, and the fusion branch, respectively. Guided by the training strategy of competitive learning, the entire network is fine-tuned in the direction of the optimal fusion of the dual modalities. Extensive experiments on several publicly available RGB-T datasets show that our tracker has superior performance compared to other latest RGB-T and RGB tracking approaches.

A Survey for Deep RGBT Tracking

“…Therefore, this work can act like a reference for researchers who are interested in RGBT tracking. VOT-RGBT2019 [43] VOT-RGBT2020 [44] Trackers Accuracy(↑) Robustness(↑) EAO(↑) Accuracy(↑) Robustness(↑) EAO(↑) mfDiMP [39] 0.6019 0.8036 0.3879 0.6380 0.7930 0.3800 MANet [18] 0.5823 0.7010 0.3463 ---SiamFT [33] 0.6300 0.6390 0.3100 ---MaCNet [25] 0.5451 0.5914 0.3052 ---JMMAC [42] 0.6649 0.8211 0.4826 0.6620 0.8180 0.4200 MANet++ [22] 0.5092 0.5379 0.2716 ---TFNet [27] 0.4617 0.5936 0.2878 ---FANet [14] 0.4724 0.5078 0.2465 ---ADRNet [29] 0.6218 0.7657 0.3959 ---SiamCDA [36] 0.6820 0.7570 0.4240 ---…”

“…However, to keep the modalityspecific characteristics being discriminative, the features from RGB and TIR modalities are also retained in some methods. Specifically, MacNet [25] learns the fusion weights through the independent modal-aware attention network and competitive learning [26], for which the features of RGB and TIR modalities are reserved, leverages the capacity of results from single modality and the modality-fused branch. TFNet [27] deploys a trident branch architecture and each branch is specific for the RGB, TIR and fused features.…”

LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking

“…These methods have fast tracking speed, but are usually weak in representing low-resolution objects, which are common in RGBT tracking. The other main research stream is in MDNet frameworks [30], [31], [32], [7], [33], [34], [35], [36], [37], [38], which performs different fusion strategies to utilize complementary benefits of RGB and thermal data. Such kinds of methods receive robust tracking results but have low efficiency, and the tracking capacity is limited by MDNet which bases on VGG network.…”

Temporal Aggregation for Adaptive RGBT Tracking

“…All the deeplearning-based trackers mentioned above merely preserve the fused features or the features from each modality. To retain the original features after fusion, [43] performs multi-modal multilayer fusion with the weights computed from the original input images. Similarly, TFNet [44] uses a trident fusion network to better excavate the complementary characteristics of RGB and TIR modalities.…”