Cross-Modal Ranking with Soft Consistency and Noisy Labels for Robust RGB-T Tracking

Li, Chenglong; Zhu, Chengli; Huang, Yan; Tang, Jin; Wang, Liang

doi:10.1007/978-3-030-01261-8_49

Cited by 132 publications

(75 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The object tracking problem of RGB-T is an extension of the traditional visual tracking task, that is, given the initial position state of the target, the RGB and thermal infrared image are comprehensively used to continuously estimate the target position in subsequent scenes. In recent years, several works have been carried out on this RGB-T tracking research, and representative approaches are roughly divided into two categories: Tracking based on traditional manual features [1][2][3][4][5][6][7] and tracking based on deep learning [8][9][10][11].The former category is mostly based on theoretical frameworks such as sparse representation [2][3][4][5], correlation filtering [6], Bayesian filtering [7], and uses hand-crafted textures or local features to construct cross-modal object appearance model and state estimation methods. The latter class builds the effective model of modeling targets from massive data by exploring the powerful feature representation capabilities of deep neural networks.…”

Section: Introductionmentioning

confidence: 99%

Object Tracking in RGB-T Videos Using Modal-Aware Attention Network and Competitive Learning

Hui

Zhang

et al. 2020

Sensors

108

View full text Add to dashboard Cite

Object tracking in RGB-thermal (RGB-T) videos is increasingly used in many fields due to the all-weather and all-day working capability of the dual-modality imaging system, as well as the rapid development of low-cost and miniaturized infrared camera technology. However, it is still very challenging to effectively fuse dual-modality information to build a robust RGB-T tracker. In this paper, an RGB-T object tracking algorithm based on a modal-aware attention network and competitive learning (MaCNet) is proposed, which includes a feature extraction network, modal-aware attention network, and classification network. The feature extraction network adopts the form of a two-stream network to extract features from each modality image. The modal-aware attention network integrates the original data, establishes an attention model that characterizes the importance of different feature layers, and then guides the feature fusion to enhance the information interaction between modalities. The classification network constructs a modality-egoistic loss function through three parallel binary classifiers acting on the RGB branch, the thermal infrared branch, and the fusion branch, respectively. Guided by the training strategy of competitive learning, the entire network is fine-tuned in the direction of the optimal fusion of the dual modalities. Extensive experiments on several publicly available RGB-T datasets show that our tracker has superior performance compared to other latest RGB-T and RGB tracking approaches.

show abstract

Section: Introductionmentioning

confidence: 99%

Object Tracking in RGB-T Videos Using Modal-Aware Attention Network and Competitive Learning

Hui

Zhang

et al. 2020

Sensors

108

View full text Add to dashboard Cite

show abstract

“…In [3], they proposed a patch-based graph model to learn object feature presentation for RGB-T tracking, where the graph is optimized via weighted sparse representations that utilize multi-modality information adaptively. Li et al [4] provided a graph-based cross-modal ranking model for RGB-T tracking, in which the soft cross-modality consistency between modalities and the optimal query learning are introduced to improve the robustness.…”

Section: B Rgb-t Vision Methodsmentioning

confidence: 99%

RGB-T Image Saliency Detection via Collaborative Graph Learning

Xia

et al. 2020

IEEE Trans. Multimedia

Self Cite

161

View full text Add to dashboard Cite

Image saliency detection is an active research topic in the community of computer vision and multimedia. Fusing complementary RGB and thermal infrared data has been proven to be effective for image saliency detection. In this paper, we propose an effective approach for RGB-T image saliency detection. Our approach relies on a novel collaborative graph learning algorithm. In particular, we take superpixels as graph nodes, and collaboratively use hierarchical deep features to jointly learn graph affinity and node saliency in a unified optimization framework. Moreover, we contribute a more challenging dataset for the purpose of RGB-T image saliency detection, which contains 1000 spatially aligned RGB-T image pairs and their ground truth annotations. Extensive experiments on the public dataset and the newly created dataset suggest that the proposed approach performs favorably against the state-of-the-art RGB-T saliency detection methods.

show abstract

“…RGBT tracking receives much attention recently and becomes more and more popular [17,21,22,16,20,23]. Recent works [21,17,16] employ reconstruction residues [21,17] or classification scores [16] to guide the weights learning of modalities to achieve adaptive fusion of RGB and thermal modalities.…”

Section: Rgbt Tracking Methodsmentioning

confidence: 99%

Multi-Adapter RGBT Tracking

Zheng

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

Self Cite

144

116

View full text Add to dashboard Cite

The task of RGBT tracking aims to take the complementary advantages from visible spectrum and thermal infrared data to achieve robust visual tracking, and receives more and more attention in recent years. Existing works focus on modality-specific information integration by introducing modality weights to achieve adaptive fusion or learning robust feature representations of different modalities. Although these methods could effectively deploy the modality-specific properties, they ignore the potential values of modality-shared cues as well as instance-aware information, which are crucial for effective fusion of different modalities in RGBT tracking. In this paper, we propose a novel Multi-Adapter convolutional Network (MANet) to jointly perform modality-shared, modality-specific and instance-aware feature learning in an end-to-end trained deep framework for RGBT tracking. We design three kinds of adapters within our network. In a specific, the generality adapter is to extract shared object representations, the modality adapter aims at encoding modality-specific information to deploy their complementary advantages, and the instance adapter is to model the appearance properties and temporal variations of a certain object. Moreover, to reduce computational complexity for real-time demand of visual tracking, we design a parallel structure of generic adapter and modality adapter. Extensive experiments on two RGBT tracking benchmark datasets demonstrate the outstanding performance of the proposed tracker against other state-ofthe-art RGB and RGBT tracking algorithms.

show abstract

Cross-Modal Ranking with Soft Consistency and Noisy Labels for Robust RGB-T Tracking

Cited by 132 publications

References 26 publications

Object Tracking in RGB-T Videos Using Modal-Aware Attention Network and Competitive Learning

Object Tracking in RGB-T Videos Using Modal-Aware Attention Network and Competitive Learning

RGB-T Image Saliency Detection via Collaborative Graph Learning

Multi-Adapter RGBT Tracking

Contact Info

Product

Resources

About