End-to-End Dense Video Captioning with Masked Transformer

Abstract

Dense video captioning aims to generate text descriptions for all events in an untrimmed video. This involves both detecting and describing events. Therefore, all previous methods on dense video captioning tackle this problem by building two models, i.e. an event proposal and a captioning model, for these two sub-problems. The models are either trained separately or in alternation. This prevents direct influence of the language description to the event proposal, which is important for generating accurate descriptions. To address this problem, we propose an end-to-end transformer model for dense video captioning. The encoder encodes the video into appropriate representations. The proposal decoder decodes from the encoding with different anchors to form video event proposals. The captioning decoder employs a masking network to restrict its attention to the proposal event over the encoding feature. This masking network converts the event proposal to a differentiable mask, which ensures the consistency between the proposal and captioning during training. In addition, our model employs a self-attention mechanism, which enables the use of efficient non-recurrent structure during encoding and leads to performance improvements. We demonstrate the effectiveness of this end-to-end model on ActivityNet Captions and YouCookII datasets, where we achieved 10.12 and 6.58

Watch, Listen and Tell: Multi-Modal Weakly Supervised Dense Event Captioning

“…The lack of sound makes it significantly more difficult, if not impossible in many cases, to describe the rich flow of the story and constituent events. Armed with this intuition, we focus on dense event captioning [22,43,49] (a.k.a. densecaptioning of events in videos [20]) and endow our models with ability to utilize rich auditory signals for both event lo-calization and captioning.…”

“…Wang et al [43] present a bidirectional proposal method that effectively exploits both past and future contexts to make proposal predictions. In [49], a differentiable masking scheme is used to ensure the consistency between proposal and captioning modules. Li et al [22] propose a descriptiveness regression component to unify the event localization and sentence generation.…”

A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions

“…1 Most related to the present work are several dense captioning approaches that have been applied to instructional videos (Zhou et al, 2018b,c). Zhou et al (2018c) achieve stateof-the-art performance on the dataset we consider; their model is video-only, and combines a region proposal network (Ren et al, 2015) and a Transformer (Vaswani et al, 2017) decoder. Multimodal Video Captioning.…”

“…As a result, all our experiments operate on a subset of the YouCook2 data. While this makes direct comparison with previous and future work more difficult, our performance metrics can be viewed as lower bounds, as they are trained on less data compared to, e.g., (Zhou et al, 2018c). Unless noted otherwise, our analyses are conducted over 1.4K videos and the 10.6K annotated segments contained therein.…”

Video Question Generation via Semantic Rich Cross-Modal Self-Attention Networks Learning

“…The recent success of deep neural networks has enabled end-to-end training in various video understanding tasks such as action recognition [3,5,7,32], and video captioning [28,29,34,39], and many of the models reach signif-*: Both authors contributed equally to this research. icant performances.…”