Two-stage modality-graphs regularized manifold ranking for RGB-T tracking

Li, Chenglong; Zhu, Chengli; Zheng, Shaofei; Luo, Bin; Tang, Jing

doi:10.1016/j.image.2018.08.004

Cited by 28 publications

(8 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…RGB-T object tracking, which is a sub-branch of visual object tracking, aims to aggregate visible and thermal infrared images for robust object tracking in challenging conditions such as low illumination, heavy occlusion and significant appearance changes [8,20]. Existing methods focus on robust target representation via integrating multi-modal source data [10,12,13,22]. One research stream aims to reconstruct the target descriptor via extracting effective features from multi-modal data.…”

Section: Feature Aggregation Methods For Rgb-t Object Trackingmentioning

confidence: 99%

SCA-MMA: Spatial and Channel-Aware Multi-Modal Adaptation for Robust RGB-T Object Tracking

et al. 2022

View full text Add to dashboard Cite

The RGB and thermal (RGB-T) object tracking task is challenging, especially with various target changes caused by deformation, abrupt motion, background clutter and occlusion. It is critical to employ the complementary nature between visual RGB and thermal infrared data. In this work, we address the RGB-T object tracking task with a novel spatial- and channel-aware multi-modal adaptation (SCA-MMA) framework, which builds an adaptive feature learning process for better mining this object-aware information in a unified network. For each type of modality information, the spatial-aware adaptation mechanism is introduced to dynamically learn the location-based characteristics of specific tracking objects at multiple convolution layers. Further, the channel-aware multi-modal adaptation mechanism is proposed to adaptively learn the feature fusion/aggregation of different modalities. In order to perform object tracking, we employ a binary classification module with two fully connected layers to predict the bounding boxes of specific targets. Comprehensive evaluations on GTOT and RGBT234 datasets demonstrate the significant superiority of our proposed SCA-MMA for robust RGB-T object tracking tasks. In particular, the precision rate (PR) and success rate (SR) on GTOT and RGBT234 datasets can reach 90.5%/73.2% and 80.2%/56.9%, significantly higher than the state-of-the-art algorithms.

show abstract

Section: Feature Aggregation Methods For Rgb-t Object Trackingmentioning

confidence: 99%

SCA-MMA: Spatial and Channel-Aware Multi-Modal Adaptation for Robust RGB-T Object Tracking

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Li et al [18] propose a crossmodal ranking method with soft consistency and noisy labels to handle the effects of modal heterogeneity and seed noises in ranking model. In addition, Li et al [19] propose a two-stage modality-graphs regularized manifold ranking model to refine the ranking results using a two-stage way.…”

Section: B Rgbt Tracking Methodsmentioning

confidence: 99%

LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking

Li,

Xue,

Jia

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

RGBT tracking receives a surge of interest in the computer vision community, but this research field lacks a large-scale and high-diversity benchmark dataset, which is essential for both the training of deep RGBT trackers and the comprehensive evaluation of RGBT tracking methods. To this end, we present a Large-scale High-diversity benchmark for RGBT tracking (LasHeR) in this work. LasHeR consists of 1224 visible and thermal infrared video pairs with more than 730K frame pairs in total. Each frame pair is spatially aligned and manually annotated with a bounding box, making the dataset well and densely annotated. LasHeR is highly diverse capturing from a broad range of object categories, camera viewpoints, scene complexities and environmental factors across seasons, weathers, day and night. We conduct a comprehensive performance evaluation of 12 RGBT tracking algorithms on the LasHeR dataset and present detailed analysis to clarify the research room in RGBT tracking. In addition, we release the unaligned version of LasHeR to attract the research interest for alignment-free RGBT tracking, which is a more practical task in real-world applications. The datasets and evaluation protocols are available at: https://github.com/BUGPLEASEOUT/LasHeR.

show abstract

“…Cvejic et al [14] investigates the effect of pixel-level fusion of visible and infrared videos on object tracking performance. After that, the representative works are based on sparse representation [15], [1], [16], [17], manifold ranking [18], [19] and dynamic graph [20], [21]. Early works focus on the sparse representation due to their robustness to noise and outliers.…”

Section: A Traditional Methods For Rgbt Trackingmentioning

confidence: 99%

“…For example, Li et al [18] propose a cross-modal manifold ranking algorithm with soft consistency and noise labels to compute the patch weights. Also, Li et al [19] propose a two-stage modality-graphs regularized manifold ranking algorithm to mitigate the impact of inaccurate patch weights initialization. These works, however, rely on the structure-fixed graphs, and the relations among patches are not well explored.…”

Section: A Traditional Methods For Rgbt Trackingmentioning

confidence: 99%

RGBT Tracking via Multi-Adapter Network with Hierarchical Divergence Loss

Lu,

Li,

Yan

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

RGBT tracking has attracted increasing attention since RGB and thermal infrared data have strong complementary advantages, which could make trackers all-day and all-weather work. Existing works usually focus on extracting modality-shared or modality-specific information, but the potentials of these two cues are not well explored and exploited in RGBT tracking. In this paper, we propose a novel multi-adapter network to jointly perform modality-shared, modality-specific and instance-aware target representation learning for RGBT tracking. To this end, we design three kinds of adapters within an end-to-end deep learning framework. In specific, we use the modified VGG-M as the generality adapter to extract the modality-shared target representations. To extract the modality-specific features while reducing the computational complexity, we design a modality adapter, which adds a small block to the generality adapter in each layer and each modality in a parallel manner. Such a design could learn multilevel modality-specific representations with a modest number of parameters as the vast majority of parameters are shared with the generality adapter. We also design instance adapter to capture the appearance properties and temporal variations of a certain target. Moreover, to enhance the shared and specific features, we employ the loss of multiple kernel maximum mean discrepancy to measure the distribution divergence of different modal features and integrate it into each layer for more robust representation learning. Extensive experiments on two RGBT tracking benchmark datasets demonstrate the outstanding performance of the proposed tracker against the state-of-the-art methods.

show abstract

Two-stage modality-graphs regularized manifold ranking for RGB-T tracking

Cited by 28 publications

References 37 publications

SCA-MMA: Spatial and Channel-Aware Multi-Modal Adaptation for Robust RGB-T Object Tracking

SCA-MMA: Spatial and Channel-Aware Multi-Modal Adaptation for Robust RGB-T Object Tracking

LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking

RGBT Tracking via Multi-Adapter Network with Hierarchical Divergence Loss

Contact Info

Product

Resources

About