Streamlined Dense Video Captioning

Abstract

Dense video captioning is an extremely challenging task since accurate and coherent description of events in a video requires holistic understanding of video contents as well as contextual reasoning of individual events. Most existing approaches handle this problem by first detecting event proposals from a video and then captioning on a subset of the proposals. As a result, the generated sentences are prone to be redundant or inconsistent since they fail to consider temporal dependency between events. To tackle this challenge, we propose a novel dense video captioning framework, which models temporal dependency across events in a video explicitly and leverages visual and linguistic context from prior events for coherent storytelling. This objective is achieved by 1) integrating an event sequence generation network to select a sequence of event proposals adaptively, and 2) feeding the sequence of event proposals to our sequential video captioning network, which is trained by reinforcement learning with two-level rewards-at both event and episode levels-for better context modeling. The proposed technique achieves outstanding performances on Ac-tivityNet Captions dataset in most metrics.

Automatic Image and Video Caption Generation With Deep Learning: A Concise Review and Algorithmic Overlap

“…However, the language model still has a large performance gap from humans in cases such as small object recognition or object recognition at lower resolutions. Similar to Krishna et al [27] work, Mun et al [64] proposed a dense video captioning framework, that models temporal dependency across events in a video explicitly and leverages visual and linguistic context from prior events for coherent storytelling. They have used Single-Stream Temporal Action model to get some proposals at a single scan, then by implying PtrNet, the highly correlated events that makeup an episode fed into a sequentional captioning network to produce a caption by RNN systems.…”

Video distillation

“…ü Ð þAE ½ü . 8 [18, 24, 25, 58, 194, 201, 206, 210, 216, 230∼241] VideoLab [241] TDDF [240] DenseVidCap [238] PickNet [234] HRL [216] RecNet [236] MMM [237] GRU-EVE [18] MA-RNN [230] OA-BTG [239] SibNet [232] Year FGM [194] LSTM-YT [206] TA [58] S2VT [24] h-RNN [231] HRNE [25] GRU-RCN [233] LSTM-TSA [58] SCN-LSTM [235] BAE [201] PickNet [234] RecNet [236] CAM-RNN [210] GRU-EVE [18] SibNet ü ý µ ü Áà [156] , ü ½ü ´¨Þ þÒµ [243] .…”

Video content analysis and retrieval system using video storytelling and indexing techniques

“…In this study, the video facts are compared with the subtitle sentence by referencing the noun of the sentence in one of the frames of a video. In [22], a new dense framework for video captioning was proposed, which specifically models the temporal dependency of occurrences in the video and utilizes visual and language contexts for coherent storytelling from past events. From the detailed literature survey, it is observed that video storytelling technique-based video content analysis and retrieval is not addressed.…”