Large Margin Structured Convolution Operator for Thermal Infrared Object Tracking

Abstract

Compared with visible object tracking, thermal infrared (TIR) object tracking can track an arbitrary target in total darkness since it cannot be influenced by illumination variations. However, there are many unwanted attributes that constrain the potentials of TIR tracking, such as the absence of visual color patterns and low resolutions. Recently, structured output support vector machine (SOSVM) and discriminative correlation filter (DCF) have been successfully applied to visible object tracking, respectively. Motivated by these, in this paper, we propose a large margin structured convolution operator (LM-SCO) to achieve efficient TIR object tracking. To improve the tracking performance, we employ the spatial regularization and implicit interpolation to obtain continuous deep feature maps, including deep appearance features and deep motion features, of the TIR targets. Finally, a collaborative optimization strategy is exploited to significantly update the operators. Our approach not only inherits the advantage of the strong discriminative capability of SOSVM but also achieves accurate and robust tracking with higher-dimensional features and more dense samples. To the best of our knowledge, we are the first to incorporate the advantages of DCF and SOSVM for TIR object tracking. Comprehensive evaluations on two thermal infrared tracking benchmarks, i.e. VOT-TIR2015 and VOT-TIR2016, clearly demonstrate that our LMSCO tracker achieves impressive results and outperforms most state-of-the-art trackers in terms of accuracy and robustness with sufficient frame rate.

LSOTB-TIR: A Large-Scale High-Diversity Thermal Infrared Object Tracking Benchmark

“…MCFTS [39] combines multiple convolutional features of VGGNet [49] with the Correlation Filters (CFs) [27] to construct an ensemble TIR tracker. Gao et al [20] combine deep appearance features [49] and deep motion features [22] with SSVM for TIR object tracking. ECO-stir [61] trains a Siamese network on synthetic TIR images to extract TIR features and then integrates them into the ECO [9] tracker.…”

“…It is widely used in video surveillance, maritime rescue, and driver assistance at night [18] since it can track the object in total darkness. In the past several years, some TIR object tracking methods [20,23,34,39,60,61] are proposed. Despite much progress, TIR object tracking faces many unsolved problems, such as distractor, intensity variation, and thermal crossover [37].…”

“…Recently, motivated by the success of deep learning in most visual tasks, several attempts [20,34,38,39] incorporate deep feature models for TIR object tracking and achieve some success. However, the used deep feature models are learned from RGB images, and we find by experiments that these RGB based deep feature models Comparison of currently widely used TIR object tracking benchmarks, including LTIR [3], VOT-TIR16 [17], PTB-TIR [37], RGB-T [31], and LSOTB-TIR.…”

Deep Learning on Multi Sensor Data for Counter UAV Applications—A Systematic Review

“…Gao et al [117] proposed a Large Margin Structured Convolutional Operator (LM-SCO) to achieve efficient object tracking based on thermal imaging. Pre-trained CNNs with RGB images were re-purposed to extract deep appearance and motion features of thermal images, which were later fused within the tracking framework.…”

Multi-Task Driven Feature Models for Thermal Infrared Tracking

“…These methods can be roughly divided into two categories, deep feature based TIR trackers and matching-based deep TIR trackers. Deep feature based TIR trackers, e.g., DSSTtir (Gundogdu et al 2016), MCFTS (Liu et al 2017), and LMSCO (Gao et al 2018), use a pre-trained classification network for extracting deep features and then integrate them into conventional trackers. Despite the demonstrated success, their performance is limited by the pre-trained deep features which are learned from RGB images and are less effective in representing TIR objects.…”