Interpretable Video Captioning via Trajectory Structured Localization

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Wang

et al. 2019

209

Typical techniques for video captioning follow the encoder-decoder framework, which can only focus on one source video being processed. A potential disadvantage of such design is that it cannot capture the multiple visual context information of a word appearing in more than one relevant videos in training data. To tackle this limitation, we propose the Memory-Attended Recurrent Network (MARN) for video captioning, in which a memory structure is designed to explore the full-spectrum correspondence between a word and its various similar visual contexts across videos in training data. Thus, our model is able to achieve a more comprehensive understanding for each word and yield higher captioning quality. Furthermore, the built memory structure enables our method to model the compatibility between adjacent words explicitly instead of asking the model to learn implicitly, as most existing models do. Extensive validation on two real-word datasets demonstrates that our MARN consistently outperforms state-of-the-art methods.

Section: Related Workmentioning

confidence: 99%

Section: Comparison On Msvdmentioning

confidence: 99%

Memory-Attended Recurrent Network for Video Captioning

Pei

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Wang

et al. 2019

209

“…Obviously, under the predefined template with fixed syntactic structure, those methods are hard to generate flexible language descriptions. Nowadays, benefit from the success of CNN and RNN, the sequence learning methods [42,53,27,28,8,45,49] are widely used to describe video content with flexible syntactic structure. In [43], Venugopalan et al obtained video representation by averaging CNN feature of each frame, which ignored the temporal information.…”

Section: Video Captioningmentioning

confidence: 99%

Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion Network

Wang

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

et al. 2019

162

In this paper, we propose to guide the video caption generation with Part-of-Speech (POS) information, based on a gated fusion of multiple representations of input videos. We construct a novel gated fusion network, with one particularly designed cross-gating (CG) block, to effectively encode and fuse different types of representations, e.g., the motion and content features of an input video. One POS sequence generator relies on this fused representation to predict the global syntactic structure, which is thereafter leveraged to guide the video captioning generation and control the syntax of the generated sentence. Specifically, a gating strategy is proposed to dynamically and adaptively incorporate the global syntactic POS information into the decoder for generating each word. Experimental results on two benchmark datasets, namely MSR-VTT and MSVD, demonstrate that the proposed model can well exploit complementary information from multiple representations, resulting in improved performances. Moreover, the generated global POS information can well capture the global syntactic structure of the sentence, and thus be exploited to control the syntactic structure of the description. Such POS information not only boosts the video captioning performance but also improves the diversity of the generated captions. Our code is at: https://github.com/vsislab/ Controllable_XGating. * This work was done while Bairui Wang was a Research Intern with Tencent AI Lab. † Corresponding authors.

“…OA-BTG constructs bidirectional temporal graph and performs object-aware feature aggregation to achieve above goals, which helps to generate accurate and fine-grained captions for better performance. TSA-ED [34] also utilizes trajectory information, which introduces a trajectory structured attentional encoderdecoder network which explores the fine-grained motion information. Although it extracts dense point trajectories, it loss the object-aware information.…”

Section: Training Detailsmentioning

confidence: 99%

Object-Aware Aggregation With Bidirectional Temporal Graph for Video Captioning

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Peng

2019

168

Video captioning aims to automatically generate natural language descriptions of video content, which has drawn a lot of attention recent years. Generating accurate and finegrained captions needs to not only understand the global content of video, but also capture the detailed object information. Meanwhile, video representations have great impact on the quality of generated captions. Thus, it is important for video captioning to capture salient objects with their detailed temporal dynamics, and represent them using discriminative spatio-temporal representations. In this paper, we propose a new video captioning approach based on object-aware aggregation with bidirectional temporal graph (OA-BTG), which captures detailed temporal dynamics for salient objects in video, and learns discriminative spatio-temporal representations by performing object-aware local feature aggregation on detected object regions. The main novelties and advantages are: (1) Bidirectional temporal graph: A bidirectional temporal graph is constructed along and reversely along the temporal order, which provides complementary ways to capture the temporal trajectories for each salient object. (2) Object-aware aggregation: Learnable VLAD (Vector of Locally Aggregated Descriptors) models are constructed on object temporal trajectories and global frame sequence, which performs object-aware aggregation to learn discriminative representations. A hierarchical attention mechanism is also developed to distinguish different contributions of multiple objects. Experiments on two widely-used datasets demonstrate our OA-BTG achieves state-of-the-art performance in terms of BLEU@4, METEOR and CIDEr metrics.