Siamese Natural Language Tracker: Tracking by Natural Language Descriptions with Siamese Trackers

Feng, Qianjin; Ablavsky, Vitaly; Bai, Qinxun; Sclaroff, Stan

doi:10.1109/cvpr46437.2021.00579

Cited by 44 publications

(37 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To mitigate the gap, we propose an asymmetrical searching strategy (ASS) to adapt the unified VL representation for improvements. Different from current VL tracking [19] adopting symmetrical and fixed template and search branches as in vision-only Siamese tracking [30], we argue that the learning framework of mixed modality should be adaptive and not fixed. To this end, ASS borrows the idea from neural architecture search (NAS) [62,40] to separately learn distinctive and asymmetrical networks for mixed modality in different branches and ModaMixers.…”

mentioning

confidence: 74%

“…Vision-Language Tracking. Natural language contains high-level semantics and has been leveraged to foster vision-related tasks [20,29,2] including tracking [32,18,19]. The work [32] first introduces linguistic description to tracking and shows that language enhances the robustness of vision-based method.…”

Section: Related Workmentioning

confidence: 99%

“…In fact, language, an equally important cue as vision, has been largely explored in vision-related tasks, and is not new to tracking. Prior works [19,32,18] have exploited vision-language (VL) multimodal learning for improving tracking. However, the performance falls far behind current SOTAs.…”

mentioning

confidence: 99%

“…However, the performance falls far behind current SOTAs. For instance on LaSOT [16], the gap between current best VL tracker [19] and recent Transformer tracker [7] is absolute 10.9% in SUC (see Fig. 1).…”

mentioning

confidence: 99%

“…Although this fusion may easily get two modalities connected, it does not accords with human learning procedure that integrates multisensory by various neurons before causal inference [3], resulting in a lower upper-bound for VL tracking. Besides, current VL trackers treat template and search branches as homoplasmic inputs, and adopt symmetrical feature learning structures for these two branches, inherited from typical vision-only Siamese tracking [19].…”

mentioning

confidence: 99%

See 4 more Smart Citations

Divert More Attention to Vision-Language Tracking

Guo¹,

Zhang²,

Fan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Relying on Transformer for complex visual feature learning, object tracking has witnessed the new standard for state-of-the-arts (SOTAs). However, this advancement accompanies by larger training data and longer training period, making tracking increasingly expensive. In this paper, we demonstrate that the Transformer-reliance is not necessary and the pure ConvNets are still competitive and even better yet more economical and friendly in achieving SOTA tracking. Our solution is to unleash the power of multimodal vision-language (VL) tracking, simply using ConvNets. The essence lies in learning novel unified-adaptive VL representations with our modality mixer (ModaMixer) and asymmetrical ConvNet search. We show that our unified-adaptive VL representation, learned purely with the ConvNets, is a simple yet strong alternative to Transformer visual features, by unbelievably improving a CNN-based Siamese tracker by 14.5% in SUC on challenging LaSOT (50.7%→65.2%), even outperforming several Transformer-based SOTA trackers. Besides empirical results, we theoretically analyze our approach to evidence its effectiveness. By revealing the potential of VL representation, we expect the community to divert more attention to VL tracking and hope to open more possibilities for future tracking beyond Transformer. Code and models will be released at https://github.com/JudasDie/SOTS.

show abstract

mentioning

confidence: 74%

Section: Related Workmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Divert More Attention to Vision-Language Tracking

Guo¹,

Zhang²,

Fan³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Deep learning‐based markerless lung tumor tracking in stereotactic radiotherapy using Siamese networks

et al. 2023

View full text Add to dashboard Cite

BackgroundRadiotherapy (RT) is involved in about 50% of all cancer patients, making it a very important treatment modality. The most common type of RT is external beam RT, which consists of delivering the radiation to the tumor from outside the body. One novel treatment delivery method is volumetric modulated arc therapy (VMAT), where the gantry continuously rotates around the patient during the radiation delivery.PurposeAccurate tumor position monitoring during stereotactic body radiotherapy (SBRT) for lung tumors can help to ensure that the tumor is only irradiated when it is inside the planning target volume. This can maximize tumor control and reduce uncertainty margins, lowering organ‐at‐risk dose. Conventional tracking methods are prone to errors, or have a low tracking rate, especially for small tumors that are in close vicinity to bony structures.MethodsWe investigated patient‐specific deep Siamese networks for real‐time tumor tracking, during VMAT. Due to lack of ground truth tumor locations in the kilovoltage (kV) images, each patient‐specific model was trained on synthetic data (DRRs), generated from the 4D planning CT scans, and evaluated on clinical data (x‐rays). Since there are no annotated datasets with kV images, we evaluated the model on a 3D printed anthropomorphic phantom but also on six patients by computing the correlation coefficient with the breathing‐related vertical displacement of the surface‐mounted marker (RPM). For each patient/phantom, we used 80% of DRRs for training and 20% for validation.ResultsThe proposed Siamese model outperformed the conventional benchmark template matching‐based method (RTR): (1) when evaluating both methods on the 3D phantom, the Siamese model obtained a 0.57–0.79‐mm mean absolute distance to the ground truth tumor locations, compared to 1.04–1.56 mm obtained by RTR; (2) on patient data, the Siamese‐determined longitudinal tumor position had a correlation coefficient of 0.71–0.98 with the RPM, compared to 0.07–0.85 for RTR; (3) the Siamese model had a 100% tracking rate, compared to 62%–82% for RTR.ConclusionsBased on these results, we argue that Siamese‐based real‐time 2D markerless tumor tracking during radiation delivery is possible. Further investigation and development of 3D tracking is warranted.

show abstract

SemTrack: A Large-Scale Dataset for Semantic Tracking in the Wild

Wang,

Hui,

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Siamese Natural Language Tracker: Tracking by Natural Language Descriptions with Siamese Trackers

Cited by 44 publications

References 34 publications

Divert More Attention to Vision-Language Tracking

Divert More Attention to Vision-Language Tracking

Deep learning‐based markerless lung tumor tracking in stereotactic radiotherapy using Siamese networks

SemTrack: A Large-Scale Dataset for Semantic Tracking in the Wild

Contact Info

Product

Resources

About