Cross-Modal Object Tracking: Modality-Aware Representations and a Unified Benchmark

Abstract

In many visual systems, visual tracking often bases on RGB image sequences, in which some targets are invalid in low-light conditions, and tracking performance is thus affected significantly. Introducing other modalities such as depth and infrared data is an effective way to handle imaging limitations of individual sources, but multi-modal imaging platforms usually require elaborate designs and cannot be applied in many real-world applications at present. Near-infrared (NIR) imaging becomes an essential part of many surveillance cameras, whose imaging is switchable between RGB and NIR based on the light intensity. These two modalities are heterogeneous with very different visual properties and thus bring big challenges for visual tracking. However, existing works have not studied this challenging problem. In this work, we address the cross-modal object tracking problem and contribute a new video dataset, including 654 cross-modal image sequences with over 481K frames in total, and the average video length is more than 735 frames. To promote the research and development of cross-modal object tracking, we propose a new algorithm, which learns the modality-aware target representation to mitigate the appearance gap between RGB and NIR modalities in the tracking process. It is plug-and-play and could thus be flexibly embedded into different tracking frameworks. Extensive experiments on the dataset are conducted, and we demonstrate the effectiveness of the proposed algorithm in two representative tracking frameworks against 19 state-of-the-art tracking methods. Dataset, code, model and results are available at https://github.com/mmic-lcl/source-code.

Cross-Modal Object Tracking: Modality-Aware Representations and A Unified Benchmark

“…In Fig. 3, we present the results of adaptive weight prediction for a typical sequence, comparing them with the tracking results of the baseline tracker DiMP [29] and MArMOT [12]. It is evident that our method demonstrates high accuracy in both fusion weight prediction and tracking results, thereby validating the effectiveness of our approach.…”

“…Then, we embed MAFM to bridge the appearance gap between RGB and NIR modalities by learning modality-aware fusion target representations through the proposed adaptive weighting mechanism. Finally, [29] and MArMOT [12]. the modality-aware fusion feature is sent to the classification branch and regression branch to perform target localization.…”

“…The evaluated trackers in our study can be divided into three types. The first type consists of 15 matching-based two-stream methods, including MArMOT DiMP [12], ToMP [37], TrDiMP [38], TransT [39], Ocean [40], SiamBAN [41], TACT [42], DiMP [29], ATOM [43], SiamRPN++ [16], SiamMask [44], SiamDW [18], GradNet [45], DaSiamRPN [17], and SiamFC [15]. The second type includes 4 classification-based methods, namely MArMOT RT [12], RT-MDNet [14] 1 , VITAL [47], and MD-Net [13].…”

“…• We conduct extensive experiments to demonstrate the superior performance of our proposed method compared to state-of-the-art trackers. This paper is an extended version of our conference paper [12] and brings three main improvements. Firstly, we propose a novel method called MAFNet, which enhances the training algorithm compared to the previous MArMOT [12].…”

“…This paper is an extended version of our conference paper [12] and brings three main improvements. Firstly, we propose a novel method called MAFNet, which enhances the training algorithm compared to the previous MArMOT [12]. It achieves modality-specific representation learning and modality-aware fusion in an end-to-end training manner, eliminating the need for tedious multi-stage training methods and addressing the issue of sub-optimization.…”