Spatio-Temporal Attention Models for Grounded Video Captioning

Abstract

Abstract. Automatic video captioning is challenging due to the complex interactions in dynamic real scenes. A comprehensive system would ultimately localize and track the objects, actions and interactions present in a video and generate a description that relies on temporal localization in order to ground the visual concepts. However, most existing automatic video captioning systems map from raw video data to high level textual description, bypassing localization and recognition, thus discarding potentially valuable information for content localization and generalization. In this work we present an automatic video captioning model that combines spatio-temporal attention and image classification by means of deep neural network structures based on long short-term memory. The resulting system is demonstrated to produce state-of-the-art results in the standard YouTube captioning benchmark while also offering the advantage of localizing the visual concepts (subjects, verbs, objects), with no grounding supervision, over space and time.

Grounded Video Description

“…They first use offthe-shelf or fine-tuned object detectors to propose object proposals/detections as for the visual recognition heavylifting. Then, in the second stage, they either attend to the object regions dynamically [17,39,1] or classify the regions into labels and fill into pre-defined/generated sentence templates [16,5,13]. However, directly generating proposals from off-the-shelf detectors causes the proposals to bias towards classes in the source dataset (i.e.…”

Adversarial Inference for Multi-Sentence Video Description

“…Over the past years there has been an increased interest in video description generation, notably with the broader adoption of the deep learning techniques. S2VT [58] was among the first approaches based on LSTMs [19,11]; some of the later ones include [38,49,52,68,72,73]. Most recently, a number of approaches to video description have been proposed, such as replacing LSTM with a Transformer Network [76], introducing a reconstruction objective [59], using bidirectional attention fusion for context modeling [61], and others [7,13,33].…”

Generating Descriptions with Grounded and Co-referenced People

“…Typically the focus is to ge-nerate a single sentence about a single image [7,20,28,52,55], video [7,13,35,39,48], or most closely to this work, movie clip [36,51]. Several works also produce grounding while generating the description: [55] propose an attention mechanism to ground each word to spatial CNN image features, [57] extend this to bounding boxes, [56] to video frames, and [61] to spatial-temporal proposals. [25] look into evaluating attention correctness for image captioning.…”