Fast RGB-T Tracking via Cross-Modal Correlation Filters

Zhai, Sulan; Shao, Pengpeng; Liang, Xinyan; Wang, Xin

doi:10.1016/j.neucom.2019.01.022

Cited by 59 publications

(14 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because the fusion of RGB and TIR data more easily achieves all-weather object tracking in the open environment, the researches on RGB-T object tracking methods become more and more popular. From the perspective of data fusion, the RGB-T object tracking framework can be roughly divided into traditional methods [ 38 , 39 ], sparse representation (SR)-based [ 40 , 41 , 42 , 43 , 44 ], graph-based [ 45 , 46 , 47 ], correlation filter-based [ 48 , 49 , 50 , 51 ], and deep learning-based approaches. Earlier studies used manual features to perform the appearance modeling of the target object.…”

Section: Related Workmentioning

confidence: 99%

Channel Exchanging for RGB-T Tracking

Zhao

Zhu

Ren

et al. 2021

Sensors

View full text Add to dashboard Cite

It is difficult to achieve all-weather visual object tracking in an open environment only utilizing single modality data input. Due to the complementarity of RGB and thermal infrared (TIR) data in various complex environments, a more robust object tracking framework can be obtained using video data of these two modalities. The fusion methods of RGB and TIR data are the core elements to determine the performance of the RGB-T object tracking method, and the existing RGB-T trackers have not solved this problem well. In order to solve the current low utilization of information intra single modality in aggregation-based methods and between two modalities in alignment-based methods, we used DiMP as the baseline tracker to design an RGB-T object tracking framework channel exchanging DiMP (CEDiMP) based on channel exchanging. CEDiMP achieves dynamic channel exchanging between sub-networks of different modes hardly adding any parameters during the feature fusion process. The expression ability of the deep features generated by our data fusion method based on channel exchanging is stronger. At the same time, in order to solve the poor generalization ability of the existing RGB-T object tracking methods and the poor ability in the long-term object tracking, more training of CEDiMP on the synthetic dataset LaSOT-RGBT is added. A large number of experiments demonstrate the effectiveness of the proposed model. CEDiMP achieves the best performance on two RGB-T object tracking benchmark datasets, GTOT and RGBT234, and performs outstandingly in the generalization testing.

show abstract

Section: Related Workmentioning

confidence: 99%

Channel Exchanging for RGB-T Tracking

Zhao

Zhu

Ren

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…In the last few years, fusion tracking has attracted a lot of interests and an increasing number of researches have been published in high-quality journals and well-known conferences [2], [6], [8], [9], [30]- [38]. Generally speaking, deep learning-based methods and CF-based methods are two main kinds of fusion tracking methods nowadays.…”

Section: Rgb-infrared Fusion Trackingmentioning

confidence: 99%

“…To the best of our knowledge, Wang et al [7] presented the first CF-based fusion tracking work. After this, Zhai et al [8] proposed an RGB-infrared tracking via cross-modal correlation filters. Yun et al [38] also presented a CF-based fusion tracker.…”

Section: Rgb-infrared Fusion Trackingmentioning

confidence: 99%

“…On one hand, some of the trackers can run at high speed, but the tracking precision is not good enough. For instance, Zhai et al [8] proposed a fusion tracking method based on correlation filters whose frame rate was 224 frames per second (FPS). However, its tracking precision and robustness were not as good as the state-of-the-art trackers.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SiamFT: An RGB-Infrared Fusion Tracking Method via Fully Convolutional Siamese Networks

Zhang

Peng

et al. 2019

IEEE Access

View full text Add to dashboard Cite

Object tracking based on visible images may fail when the visible images are unreliable, for example when the illumination condition is poor. Infrared images reveal thermal radiation of objects and are insensitive to these factors. Due to the complementary features of visible and infrared images, RGB-infrared fusion tracking has attracted widespread attention recently. In this paper, an RGB-infrared fusion tracking method based on the fully convolutional Siamese Networks, termed as SiamFT, is proposed. Visible and infrared images are firstly processed by two Siamese Networks, namely visible network and infrared network, respectively. Then, convolutional features of visible and infrared template images extracted from two Siamese Networks are concatenated to form fused template image. Convolutional features of visible and infrared search image are fused through the proposed feature fusion network adaptively. In particular, a modality weight computation method based on the response value of Siamese network is proposed to predict the reliability of different images. Cross-relation is then applied to the fused template feature and the fused search feature to produce the final response map, based on which the tracking results can be obtained. Extensive experiments indicate that the proposed SiamFT shows better performance than thestate-of-art fusion tracking algorithms at real-time speed. INDEX TERMS Deep learning, fusion tracking, object tracking, Siamese network.

show abstract

“…For example, Chen et al [1] added texture, point density, local surface, and open source features, while Liu et al [8] added depth information to improve photogrammetric point cloud segmentation. Researchers have also improved segmentation by adding thermal information [7,9].…”

Section: Introductionmentioning

confidence: 99%

A Novel Building Temperature Simulation Approach Driven by Expanding Semantic Segmentation Training Datasets with Synthetic Aerial Thermal Images

2021

View full text Add to dashboard Cite

Multi-sensor imagery data has been used by researchers for the image semantic segmentation of buildings and outdoor scenes. Due to multi-sensor data hunger, researchers have implemented many simulation approaches to create synthetic datasets, and they have also synthesized thermal images because such thermal information can potentially improve segmentation accuracy. However, current approaches are mostly based on the laws of physics and are limited to geometric models’ level of detail (LOD), which describes the overall planning or modeling state. Another issue in current physics-based approaches is that thermal images cannot be aligned to RGB images because the configurations of a virtual camera used for rendering thermal images are difficult to synchronize with the configurations of a real camera used for capturing RGB images, which is important for segmentation. In this study, we propose an image translation approach to directly convert RGB images to simulated thermal images for expanding segmentation datasets. We aim to investigate the benefits of using an image translation approach for generating synthetic aerial thermal images and compare those approaches with physics-based approaches. Our datasets for generating thermal images are from a city center and a university campus in Karlsruhe, Germany. We found that using the generating model established by the city center to generate thermal images for campus datasets performed better than using the latter to generate thermal images for the former. We also found that using a generating model established by one building style to generate thermal images for datasets with the same building styles performed well. Therefore, we suggest using training datasets with richer and more diverse building architectural information, more complex envelope structures, and similar building styles to testing datasets for an image translation approach.

show abstract

Fast RGB-T Tracking via Cross-Modal Correlation Filters

Cited by 59 publications

References 17 publications

Channel Exchanging for RGB-T Tracking

Channel Exchanging for RGB-T Tracking

SiamFT: An RGB-Infrared Fusion Tracking Method via Fully Convolutional Siamese Networks

A Novel Building Temperature Simulation Approach Driven by Expanding Semantic Segmentation Training Datasets with Synthetic Aerial Thermal Images

Contact Info

Product

Resources

About