Generating Descriptions with Grounded and Co-referenced People

Abstract

Learning how to generate descriptions of images or videos received major interest both in the Computer Vision and Natural Language Processing communities. While a few works have proposed to learn a grounding during the generation process in an unsupervised way (via an attention mechanism), it remains unclear how good the quality of the grounding is and whether it benefits the description quality. In this work we propose a movie description model which learns to generate description and jointly ground (localize) the mentioned characters as well as do visual co-reference resolution between pairs of consecutive sentences/clips. We also propose to use weak localization supervision through character mentions provided in movie descriptions to learn the character grounding. At training time, we first learn how to localize characters by relating their visual appearance to mentions in the descriptions via a semi-supervised approach. We then provide this (noisy) supervision into our description model which greatly improves its performance. Our proposed description model improves over prior work w.r.t. generated description quality and additionally provides grounding and local co-reference resolution. We evaluate it on the MPII Movie Description dataset using automatic and human evaluation measures and using our newly collected grounding and co-reference data for characters.

Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video

“…Given an image/video and a language query, image/video grounding aims to localize a spatial region in the image (Plummer et al, 2015;Yu et al, 2017Yu et al, , 2018 or a specific frame in the video (Zhou et al, 2018) which semantically corresponds to the language query. Grounding has broad applications, such as text based image retrieval (Chen et al, 2017;, description generation (Wang et al, 2018a;Rohrbach et al, 2017; A brown and white dog is lying on the grass and then it stands up.…”

Grounded Video Description

“…Visual attention usually comes in the form of temporal attention [35] (or spatial-attention [33] in the image domain), semantic attention [14,36,37,42] or both [20]. The recent unprecedented success in object detection [24,7] has regained the community's interests on detecting fine-grained visual clues while incorporating them into end-toend networks [17,27,1,16]. Description methods which are based on object detectors [17,39,1,16,5,13] tackle the captioning problem in two stages.…”

“…Instead of fine-tuning a general detector, we transfer the object classification knowledge from off-the-shelf object detectors to our model and then fine-tune this representation as part of our generation model with sparse box annotations. With a focus on co-reference resolution and identifying people, [27] proposes a framework that can refer to particular character instances and do visual co-reference resolution between video clips. However, their method is restricted to identifying human characters whereas we study more general the grounding of objects.…”

Self-Supervised Learning of Face Representations for Video Face Clustering

“…someone opens the door). However, recent work [24] suggests that more meaningful captions can be achieved from an improved understanding of characters. In general, the ability to predict which characters appear when and where facilitates a deeper video understanding that is grounded in the storyline.…”