Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking

Chen, Bo‐Yu; Li, Peixia; Bai, Lei; Qiao, Lei; Shen, Qiuhong; Li, Bo; Gan, Weihao; Wu, Wei; Ouyang, Wanli

doi:10.1007/978-3-031-20047-2_22

Cited by 119 publications

(40 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this way, our MixFormer unifies the two processes of feature extraction and information integration with an iterative MAM based backbone, leading to a more compact, neat and effective end-to-end tracker. After our CVPR 2022 conference version [16], there are some co-occurent works in ECCV 2022 such as OSTrack [82] and SimTrack [9], which also use Transformer as backbone to perform both feature extraction and information fusion. Our work starts earlier and obtains better performance than them due to our customized design of localization head, well-explored pre-trained strategies and backbone, and effective online templates selection method.…”

Section: Tracking Paradigmmentioning

confidence: 99%

MixFormer: End-to-End Tracking with Iterative Mixed Attention

Cui¹,

Cheng²,

Wu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Visual object tracking often employs a multi-stage pipeline of feature extraction, target information integration, and bounding box estimation. To simplify this pipeline and unify the process of feature extraction and target information integration, in this paper, we present a compact tracking framework, termed as MixFormer, built upon transformers. Our core design is to utilize the flexibility of attention operations, and propose a Mixed Attention Module (MAM) for simultaneous feature extraction and target information integration. This synchronous modeling scheme allows to extract target-specific discriminative features and perform extensive communication between target and search area. Based on MAM, we build our MixFormer trackers simply by stacking multiple MAMs and placing a localization head on top. Specifically, we instantiate two types of MixFormer trackers, a hierarchical tracker MixCvT, and a non-hierarchical tracker MixViT. For these two trackers, we investigate a series of pre-training methods and uncover the different behaviors between supervised pre-training and self-supervised pre-training in our MixFormer trackers. We also extend the masked pre-training to our MixFormer trackers and design the competitive TrackMAE pre-training technique. Finally, to handle multiple target templates during online tracking, we devise an asymmetric attention scheme in MAM to reduce computational cost, and propose an effective score prediction module to select high-quality templates. Our MixFormer trackers set a new state-of-the-art performance on seven tracking benchmarks, including LaSOT, TrackingNet, VOT2020, GOT-10k, OTB100 and UAV123. In particular, our MixViT-L achieves AUC score of 73.3% on LaSOT, 86.1% on TrackingNet, EAO of 0.584 on VOT2020, and AO of 75.7% on GOT-10k. Code and trained models are publicly available at https://github.com/MCG-NJU/MixFormer.

show abstract

Section: Tracking Paradigmmentioning

confidence: 99%

MixFormer: End-to-End Tracking with Iterative Mixed Attention

Cui¹,

Cheng²,

Wu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In addition, active and vibrant research has been conducted on transformer-based tracking methods that adopt a lightweight backbone for aerial tracking 54,55 . Unlike the trackers mentioned above, the research on trackers in which the backbone is replaced with transformers instead of existing CNNs also shows remarkable performance 60,61 .…”

Section: Related Work Transformer In Visual Trackingmentioning

confidence: 99%

AMST2: Aggregated Multi-Level Spatial and Temporal Context-Based Transformer for Robust Aerial Tracking

Park

Lee

Jeong

et al. 2023

Preprint

View full text Add to dashboard Cite

Recently, many existing visual trackers have made significant progress by incorporating either spatial information from multi-level convolution layers or temporal information for tracking. However, the complementary advantages of both spatial and temporal information cannot be leveraged when these two types of information are used separately. In this paper, we present a new approach for robust visual tracking using a transformer-based model that incorporates both spatial and temporal context information at multiple levels. To integrate the refined similarity maps through multi-level spatial and temporal encoders, we propose an aggregation encoder. Consequently, the output of the proposed aggregation encoder contains useful features that integrate the global contexts of multi-level spatial and the temporal contexts. The contrasting yet complementary representation of multi-level spatial and temporal contexts provided by this feature can help prevent tracking failures in a range of complex aerial scenarios, including occlusion, motion blur, and scale variations. Additionally, the proposed architecture can achieve more robust object tracking against significant variations by updating the features of the latest object while retaining the initial template information. Extensive experiments on five challenging short-term and long-term aerial tracking benchmarks have demonstrated that the proposed tracker outperforms state-of-the-art tracking methods in terms of both real-time processing speed and performance.

show abstract

“…Because transformers were originally designed for sequence-to-sequence learning on textual data and have exhibited good performance, their ability to integrate global information has been gradually unveiled and transformers have been extended to other modern deep learning applications such as image classification (Liu et al, 2020 ; Chen C. -F. R. et al, 2021 ; He et al, 2021 ), reinforcement learning (Parisotto et al, 2020 ; Chen L. et al, 2021 ), face alignment (Ning et al, 2020 ), object detection (Beal et al, 2020 ; Carion et al, 2020 ), image recognition (Dosovitskiy et al, 2020 ) and object tracking (Yan et al, 2019 , 2021a ; Cao et al, 2021 ; Lin et al, 2021 ; Zhang J. et al, 2021 ; Chen B. et al, 2022 ; Chen et al, 2022b ; Mayer et al, 2022 ). Based on CNNs and transformers, the DERT (Carion et al, 2020 ) applies a transformer to object detection tasks.…”

Section: Related Workmentioning

confidence: 99%

“…Over the past few years, visual object tracking has made significant advancements based on the development of convolutional neural networks due to the breakthroughs that have been made to generate more powerful backbones, such as deeper networks (He et al, 2016 ; Chen B. et al, 2022 ), efficient network structure (Howard et al, 2017 ), attention mechanism (Hu et al, 2018 ). Inspired by the way of the human brain process the overload information (Wolfe and Horowitz, 2004 ), the attention mechanism is utilized to enhance the vital features and surpass the unnecessary information of the input feature.…”

Section: Introductionmentioning

confidence: 99%

Siamese hierarchical feature fusion transformer for efficient tracking

Dai

Wang

et al. 2022

Front. Neurorobot.

View full text Add to dashboard Cite

Object tracking is a fundamental task in computer vision. Recent years, most of the tracking algorithms are based on deep networks. Trackers with deeper backbones are computationally expensive and can hardly meet the real-time requirements on edge platforms. Lightweight networks are widely used to tackle this issue, but the features extracted by a lightweight backbone are inadequate for discriminating the object from the background in complex scenarios, especially for small objects tracking task. In this paper, we adopted a lightweight backbone and extracted features from multiple levels. A hierarchical feature fusion transformer (HFFT) was designed to mine the interdependencies of multi-level features in a novel model—SiamHFFT. Therefore, our tracker can exploit comprehensive feature representations in an end-to-end manner, and the proposed model is capable of handling small target tracking in complex scenarios on a CPU at a rate of 29 FPS. Comprehensive experimental results on UAV123, UAV123@10fps, LaSOT, VOT2020, and GOT-10k benchmarks with multiple trackers demonstrate the effectiveness and efficiency of SiamHFFT. In particular, our SiamHFFT achieves good performance both in accuracy and speed, which has practical implications in terms of improving small object tracking performance in the real world.

show abstract

Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking

Cited by 119 publications

References 45 publications

MixFormer: End-to-End Tracking with Iterative Mixed Attention

MixFormer: End-to-End Tracking with Iterative Mixed Attention

AMST2: Aggregated Multi-Level Spatial and Temporal Context-Based Transformer for Robust Aerial Tracking

Siamese hierarchical feature fusion transformer for efficient tracking

Contact Info

Product

Resources

About