Real-time Visual Object Tracking with Natural Language Description

Abstract

Tracking with natural-language (NL) specification is a powerful new paradigm to yield trackers that initialize without a manuallyspecified bounding box, stay on target in spite of occlusions, and auto-recover when diverged. These advantages stem in part from visual appearance and NL having distinct and complementary invariance properties. However, realizing these advantages is technically challenging: the two modalities have incompatible representations. In this paper, we present the first practical and competitive solution to the challenge of tracking with NL specification. Our first novelty is an NL region proposal network (NL-RPN) that transforms an NL description into a convolutional kernel and shares the search branch with siamese trackers; the combined network can be trained end-to-end. Secondly, we propose a novel formulation to represent the history of past visual exemplars and use those exemplars to automatically reset the tracker together with our NL-RPN. Empirical results over tracking benchmarks with NL annotations demonstrate the effectiveness of our approach.

Capsule-based Object Tracking with Natural Language Specification

“…Secondly, as depicted in Figure 1, the tracker's ability to accurately locate the target in long-term scenes has been substantially improved by taking advantage of jointly optimized the two heterogeneous features of vision and language. However, the previously published methods [15,16,29] did not achieve competitive results on the popular tracking benchmarks. We found that the current method has the following two problems: (1) The learning strategy of latent alignment between the text query and the video frame is overly simplistic, and (2) the decoded representation of 2D convolution-based models has some limitations to capture the relationship between different entities and the parts of the entity itself.…”

“…Recently, tracking by natural language specification (TNL) that does not require a manually-specified bounding box for initialization has been studied [15,16,29]. Combining natural language understanding and visual object tracking has the following benefits.…”

“…In [29], a dynamic convolutional filter is incorporated to optimize the joint embedding representation of video frame and text query. Similarly, in [16], a natural language region proposal network (NL-RPN) is designed to represent the text query, and the similarity between the text query and video frame is measured by a convolutional filter. For [16,29], the generated dynamic convolutional filter is unstable as the linguistic variation of textual input description would seriously affect the language embedding.…”

“…Similarly, in [16], a natural language region proposal network (NL-RPN) is designed to represent the text query, and the similarity between the text query and video frame is measured by a convolutional filter. For [16,29], the generated dynamic convolutional filter is unstable as the linguistic variation of textual input description would seriously affect the language embedding. For a concrete example, "runner in the middle with white shirt" and "runner with white shirt in the middle", i.e., two sentences with the same meaning may produce different dynamic convolution filters, resulting in different tracking results.…”

“…For a concrete example, "runner in the middle with white shirt" and "runner with white shirt in the middle", i.e., two sentences with the same meaning may produce different dynamic convolution filters, resulting in different tracking results. The 𝑅𝑒𝑐𝑢𝑟𝑟𝑒𝑛𝑡 𝑁 𝑒𝑢𝑟𝑎𝑙 𝑁 𝑒𝑡𝑤𝑜𝑟𝑘 (RNN), as used in the prior attempt [15] to measure the similarities between visual proposals and the input language embedding, is overly simplistic. Besides, the heterogeneous feature fusion strategy in the aforementioned methods does not guarantee whether the frame features perceive the query-focused context region, which is crucial for tracking tasks.…”