Capsule-based Object Tracking with Natural Language Specification

Abstract

Tracking with Natural-Language Specification (TNL) is a joint topic of understanding the vision and natural language with a wide range of applications. In previous works, the communication between two heterogeneous features of vision and language is mainly through a simple dynamic convolution. However, the performance of prior works is capped by the difficulty of linguistic variation of natural language in modeling the dynamically changing target and its surroundings. In the meanwhile, natural language and vision are firstly fused and then utilized for tracking, which is hard to model the query-focused context. Query-focused should pay more attention to context modeling to promote the correlation between these two features. To address these issues, we propose a capsule-based network, referred to as CapsuleTNL, which performs regression tracking with natural language query. In the beginning, the visual and textual input is encoded with capsules, which can not only establish the relationship between entities but also the relationship between the parts of the entity itself. Then, we devise two interaction routing modules, which consist of visual-textual routing module to reduce the linguistic variation of input query and textual-visual routing module to precisely incorporate query-based visual cues simultaneously. To validate the potential of the proposed network for visual object tracking, we evaluate our method on two large tracking benchmarks. The experimental evaluation demonstrates the effectiveness of our capsule-based network.

School of Mechanical Engineering, Shanghai Jiao Tong University, 200240, Shanghai, China

“…In recent years, the two-stream framework [2], [4], [5], [8] has emerged as a dominant VL tracking paradigm (see Fig. 1(a)).…”

“…In the past few years, two-stream VL trackers [2], [4], [5], [8], which extract visual features and language features separately and then perform feature interaction in a fusion model (as shown in Fig 1(a)), have emerged as a domain framework and obtained significant progresses. For instance, Feng et al [4] proposed a Siamese natural language region proposal network for multi-stage feature extraction, and then applied an aggregation module to dynamically combine predictions from both visual and language modalities.…”

“…Firstly, the separation of feature extraction and integration prevents the model from performing early multi-modal feature interaction, resulting in limited objectbackground discriminative power [10], [11]. Although some works have attempted to design complicated [8] or multistage [4], [5] fusion models to enhance the associations between modalities, the lack of mutual interaction remains an insurmountable gap. More seriously, heavy fusion models increase the number of parameters, leading to significant computational inefficiency.…”