RGBT Tracking via Multi-Adapter Network with Hierarchical Divergence Loss

Abstract

RGBT tracking has attracted increasing attention since RGB and thermal infrared data have strong complementary advantages, which could make trackers all-day and all-weather work. Existing works usually focus on extracting modality-shared or modality-specific information, but the potentials of these two cues are not well explored and exploited in RGBT tracking. In this paper, we propose a novel multi-adapter network to jointly perform modality-shared, modality-specific and instance-aware target representation learning for RGBT tracking. To this end, we design three kinds of adapters within an end-to-end deep learning framework. In specific, we use the modified VGG-M as the generality adapter to extract the modality-shared target representations. To extract the modality-specific features while reducing the computational complexity, we design a modality adapter, which adds a small block to the generality adapter in each layer and each modality in a parallel manner. Such a design could learn multilevel modality-specific representations with a modest number of parameters as the vast majority of parameters are shared with the generality adapter. We also design instance adapter to capture the appearance properties and temporal variations of a certain target. Moreover, to enhance the shared and specific features, we employ the loss of multiple kernel maximum mean discrepancy to measure the distribution divergence of different modal features and integrate it into each layer for more robust representation learning. Extensive experiments on two RGBT tracking benchmark datasets demonstrate the outstanding performance of the proposed tracker against the state-of-the-art methods.

LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking

“…These methods have fast tracking speed, but are usually weak in representing low-resolution objects, which are common in RGBT tracking. The other main research stream is in MDNet frameworks [30], [31], [32], [7], [33], [34], [35], [36], [37], [38], which performs different fusion strategies to utilize complementary benefits of RGB and thermal data. Such kinds of methods receive robust tracking results but have low efficiency, and the tracking capacity is limited by MDNet which bases on VGG network.…”

“…We evaluate 12 RGBT tracking algorithms on LasHeR to provide a comprehensive platform of performance analysis. Deep RGBT trackers include MANet [7], DAPNet [33], MaC-Net [32], DAFNet [34], FANet [31], MANet++ [38], DMC-Net [37] and mfDiMP [10]. RGBT trackers based on handcrafted features include SGT [3], CMR [18] and SGT++ [4].…”

“…SGT [3] 2017 CMR [18] 2018 SGT++ [4] 2018 MANet [7] 2019 DAPNet [33] 2019 DAFNet [34] 2019 mfDiMP [10] 2019 MaCNet [32] 2020 CAT [35] 2020 FANet [31] 2020 MANet++ [38] 2020 DMCNet [37] 2020…”

“…In this part, we conduct a qualitative evaluation of nine deep trackers including MANet [7], DAPNet [33], MaCNet [32], DAFNet [34], FANet [31], MANet++ [38], DMCNet [37] and mfDiMP [10]. We select five representative sequences in LasHeR where there are frequent challenges such as total occlusion, thermal crossover, deformation, similar appearance, and visualize the tracking results of these trackers on these sequences, as shown in Fig.…”

“…In the comparison with the tracker trained on the large-scale synthetic dataset, it can be found that the improvement of PR and SR scores proves the necessity of a large-scale real but not synthetic RGBT tracking dataset. MANET [7], DAPNET [33], MACNET [32], DAFNET [34], FANET [31], CAT [35], MANET++ [38], DMCNET [37], MFDIMP [10], SGT [3], CMR [18] AND SGT++ [4]. MANET [7] AND MFDIMP [10] ON LASHER.…”