Describe and Attend to Track: Learning Natural Language guided Structural Representation and Visual Attention for Object Tracking

Wang, Xiao; Li, Chenglong; Yang, Rui; Zhang, Tianzhu; Tang, Jin; Luo, Bin

doi:10.48550/arxiv.1811.10014

Cited by 10 publications

(33 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1 (a), the tracker may be confused to track the bike or lower body of the pedestrian. Similar views can also be found in [21,43,65,78]. (3) Current BBox-based trackers may perform poorly when facing abrupt appearance variation of the target object, like face/cloth changing or species variation in Fig.…”

Section: Introductionmentioning

confidence: 71%

“…Lingual Specification Only; Lingual First, then Visual Specification; Lingual and Visual Specification). Wang [65] and Feng [21] also propose to use the language information to generate global proposals for tracking. Yang et al…”

Section: Related Workmentioning

confidence: 99%

“…Recently, some researchers attempt to introduce the natural language description instead of the BBox for tracking [21,43,65,78], termed tracking by natural language. This setting allows for a new type of human-machine interaction in object tracking.…”

Section: Introductionmentioning

confidence: 99%

“…However, this research topic has received far less attention than standard target tracking. Only a few works [20,21,43,65,78] are developed and compared on tracking benchmark datasets specially designed for BBox based tracking. These benchmarks may fail to nicely reflect the true power of tracking-by-language, and this inspires us to design a new and large-scale benchmark for this task.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark

Wang¹,

Shu²,

Zhang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Tracking by natural language specification is a new rising research topic that aims at locating the target object in the video sequence based on its language description. Compared with traditional bounding box (BBox) based tracking, this setting guides object tracking with high-level semantic information, addresses the ambiguity of BBox, and links local and global search organically together. Those benefits may bring more flexible, robust and accurate tracking performance in practical scenarios. However, existing natural language initialized trackers are developed and compared on benchmark datasets proposed for trackingby-BBox, which can't reflect the true power of trackingby-language. In this work, we propose a new benchmark specifically dedicated to the tracking-by-language, including a large scale dataset, strong and diverse baseline methods. Specifically, we collect 2k video sequences (contains a total of 1,244,340 frames, 663 words) and split 1300/700 for the train/testing respectively. We densely annotate one sentence in English and corresponding bounding boxes of the target object for each video. We also introduce two new challenges into TNL2K for the object tracking task, i.e., adversarial samples and modality switch. A strong baseline method based on an adaptive local-global-search scheme is proposed for future works to compare. We believe this benchmark will greatly boost related researches on natural language guided tracking.

show abstract

Section: Introductionmentioning

confidence: 71%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark

Wang¹,

Shu²,

Zhang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Results on LaSOT [46]: LaSOT is the largest dataset for long-term tracking which contains 1400 videos (Protocol-I and protocol-II employ all the videos and 280 videos for testing respectively). In this paper, we adopt the protocol-II for the evaluation of our tracker and compared trackers including CSRDCF [42], Lang-Tracker [94], ECO [4], DSiam [95], VITAL [35], THOR [80], MDNet [3], SiamRPN++ [13], Ocean [89], SiamFC++ [52], LTMU [65]. As shown in Fig.…”

Section: Comparison On Public Benchmarksmentioning

confidence: 99%

Tracking by Joint Local and Global Search: A Target-aware Attention based Approach

Xiao

Tang

Luo

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Tracking-by-detection is a very popular framework for single object tracking which attempts to search the target object within a local search window for each frame. Although such local search mechanism works well on simple videos, however, it makes the trackers sensitive to extremely challenging scenarios, such as heavy occlusion and fast motion. In this paper, we propose a novel and general target-aware attention mechanism (termed TANet) and integrate it with tracking-bydetection framework to conduct joint local and global search for robust tracking. Specifically, we extract the features of target object patch and continuous video frames, then we concatenate and feed them into a decoder network to generate targetaware global attention maps. More importantly, we resort to adversarial training for better attention prediction. The appearance and motion discriminator networks are designed to ensure its consistency in spatial and temporal views. In the tracking procedure, we integrate the target-aware attention with multiple trackers by exploring candidate search regions for robust tracking. Extensive experiments on both short-term and long-term tracking benchmark datasets all validated the effectiveness of our algorithm. The project page of this paper can be found at https: //sites.google.com/view/globalattentiontracking/home/extend.

show abstract

Joint visual template and natural language for robust visual tracking

Wang

Zhang

2022

Electronics Letters

View full text Add to dashboard Cite

At present, the target of interest in visual tracking is given in the form of a bounding box. Due to the randomness of the target shape, the bounding box may contain a lot of non-target information. When encountering complex tracking scenarios, the performance of the tracker reduces severely. To address this problem, in this letter, the authors propose a novel tracking framework based on the joint of the visual template and natural language (VNTrack) to alleviate the impact of bounding box ambiguity. Specifically, the authors first use a pre-trained language model to extract the features of the language description of the target. Then, a feature alignment module is designed to align and enhance the visual template feature and natural language feature. In addition, the authors design a multimodal query module to fuse the visual template, natural language, and search region information. Experimental results over tracking benchmarks with language annotations show that the proposed VNTrack is competitive among the state-of-the-art trackers. Data availability statement:The data that support the findings of this study are available from the corresponding author upon reasonable request.

show abstract

Describe and Attend to Track: Learning Natural Language guided Structural Representation and Visual Attention for Object Tracking

Cited by 10 publications

References 36 publications

Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark

Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark

Tracking by Joint Local and Global Search: A Target-aware Attention based Approach

Joint visual template and natural language for robust visual tracking

Contact Info

Product

Resources

About