LasHeR: A Large-Scale High-Diversity Benchmark for RGBT Tracking

Li, Chenglong; Xue, Wei; Jia, Yaqing; Qu, Zhichen; Luo, Bin; Tang, Jin; Sun, Dengdi

doi:10.1109/tip.2021.3130533

Cited by 117 publications

(61 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we evaluate our algorithm by comparing the tracking performance with some state-of-the-art trackers on three RGBT tracking benchmarks including GTOT (Li et al 2016), RGBT234 (Li et al 2019a) and LasHeR (Li et al 2021) to validate the effectiveness of proposed method and analyze effectiveness of each major component in the algorithm. In this part, we will introduce the details of the datasets, the evaluation metrics, and the implementation details of training and testing.…”

Section: Methodsmentioning

confidence: 99%

Attribute-Based Progressive Fusion Network for RGBT Tracking

Xiao

Yang

et al. 2022

AAAI

Self Cite

101

View full text Add to dashboard Cite

RGBT tracking usually suffers from various challenge factors, such as fast motion, scale variation, illumination variation, thermal crossover and occlusion, to name a few. Existing works often study fusion models to solve all challenges simultaneously, and it requires fusion models complex enough and training data large enough, which are usually difficult to be constructed in real-world scenarios. In this work, we disentangle the fusion process via the challenge attributes, and thus propose a novel Attribute-based Progressive Fusion Network (APFNet) to increase the fusion capacity with a small number of parameters while reducing the dependence on large-scale training data. In particular, we design five attribute-specific fusion branches to integrate RGB and thermal features under the challenges of thermal crossover, illumination variation, scale variation, occlusion and fast motion respectively. By disentangling the fusion process, we can use a small number of parameters for each branch to achieve robust fusion of different modalities and train each branch using the small training subset with the corresponding attribute annotation. Then, to adaptive fuse features of all branches, we design an aggregation fusion module based on SKNet. Finally, we also design an enhancement fusion transformer to strengthen the aggregated feature and modality-specific features. Experimental results on benchmark datasets demonstrate the effectiveness of our APFNet against other state-of-the-art methods.

show abstract

Section: Methodsmentioning

confidence: 99%

Attribute-Based Progressive Fusion Network for RGBT Tracking

Xiao

Yang

et al. 2022

AAAI

Self Cite

101

View full text Add to dashboard Cite

show abstract

“…It is considered a positive sample when the ratio of overlap rate (IoU) between the sample and the ground truth is [0.7, 1], while a sample in the range of [0, 0.5] is considered a negative sample. Notably, we train MSIFNet with the GTOT [ 43 ] dataset and test it on RGBT234 [ 44 ] and LasHeR [ 45 ] datasets. When testing on GTOT, we randomly selected 50 video sequences on RGBT234 for training.…”

Section: Methodsmentioning

confidence: 99%

Multi-Scale Feature Interactive Fusion Network for RGBT Tracking

Xiao

Xiong

Meng

et al. 2023

Sensors

View full text Add to dashboard Cite

The fusion tracking of RGB and thermal infrared image (RGBT) is paid wide attention to due to their complementary advantages. Currently, most algorithms obtain modality weights through attention mechanisms to integrate multi-modalities information. They do not fully exploit the multi-scale information and ignore the rich contextual information among features, which limits the tracking performance to some extent. To solve this problem, this work proposes a new multi-scale feature interactive fusion network (MSIFNet) for RGBT tracking. Specifically, we use different convolution branches for multi-scale feature extraction and aggregate them through the feature selection module adaptively. At the same time, a Transformer interactive fusion module is proposed to build long-distance dependencies and enhance semantic representation further. Finally, a global feature fusion module is designed to adjust the global information adaptively. Numerous experiments on publicly available GTOT, RGBT234, and LasHeR datasets show that our algorithm outperforms the current mainstream tracking algorithms.

show abstract

“…(i) For RGB-D tracking, we evaluate our tracker on Depthtrack [47] and VOT22-RGBD [16]. (ii) For RGB-T tracking, we provide the comparison results on RGBT234 [23] and LasHeR [25]. (iii) For RGB-E tracking, we report the experimental results on the largest VisEvent [43].…”

Section: Downstream Tasksmentioning

confidence: 99%

“…For example, the widely used RGB-based tracking datasets, GOT-10k [13], Track-ingNet [36], and LaSOT [9], contain 9.3K, 30.1K, and 1.1K sequences, corresponding to 1.4M, 14M, and 2.8M frames for training. Whereas the largest training datasets in multi-modal tracking, DepthTrack [47], LasHeR [25], VisEvent [43], contain 150, 979, 500 training sequences, corresponding to 0.22M, 0.51M, 0.21M annotated frame pairs, which is at least an order of magnitude less than the former. Accounting for the above limitation, multi-modal tracking methods [43,47,61] usually utilize pre-trained RGB-based trackers and perform fine-tuning on their taskoriented training sets (as shown in Figure 1 (a)→(b)).…”

Section: Introductionmentioning

confidence: 99%

Visual Prompt Multi-Modal Tracking

Zhu¹,

Lai²,

Chen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Visible-modal object tracking gives rise to a series of downstream multi-modal tracking tributaries. To inherit the powerful representations of the foundation model, a natural modus operandi for multi-modal tracking is full fine-tuning on the RGB-based parameters. Albeit effective, this manner is not optimal due to the scarcity of downstream data and poor transferability, etc. In this paper, inspired by the recent success of the prompt learning in language models, we develop Visual Prompt multi-modal Tracking (ViPT), which learns the modal-relevant prompts to adapt the frozen pre-trained foundation model to various downstream multimodal tracking tasks. ViPT finds a better way to stimulate the knowledge of the RGB-based model that is pre-trained at scale, meanwhile only introducing a few trainable parameters (less than 1% of model parameters). ViPT outperforms the full fine-tuning paradigm on multiple downstream tracking tasks including RGB+Depth, RGB+Thermal, and RGB+Event tracking. Extensive experiments show the potential of visual prompt learning for multi-modal tracking, and ViPT can achieve state-of-the-art performance while satisfying parameter efficiency. Code and models are available at https://github.com/jiawen-zhu/ViPT.

show abstract

LasHeR: A Large-Scale High-Diversity Benchmark for RGBT Tracking

Cited by 117 publications

References 64 publications

Attribute-Based Progressive Fusion Network for RGBT Tracking

Attribute-Based Progressive Fusion Network for RGBT Tracking

Multi-Scale Feature Interactive Fusion Network for RGBT Tracking

Visual Prompt Multi-Modal Tracking

Contact Info

Product

Resources

About