Sketch, Ground, and Refine: Top-Down Dense Video Captioning

Deng, Chaorui; Chen, Shizhe; Chen, Da; He, Yuan; Wu, Qi

doi:10.1109/cvpr46437.2021.00030

Cited by 57 publications

(34 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent work (Zhou et al, 2018a;Li et al, 2018;Zhou et al, 2018c;Mun et al, 2019;Iashin and Rahtu, 2020) follows the twostage "detect-then-describe" framework, in which the event proposal module first predicts a set of event segments, then the captioning module constructs captions for each candidate event segment. Another line of work (Deng et al, 2021;Wang et al, 2021) removes the explicit event proposing process. Deng et al (2021) tackles the DVC task from a top-down perspective, in which they first generate a video-level story, then ground each sentence in the story into a video segment.…”

Section: Multimodal Transformermentioning

confidence: 99%

“…Another line of work (Deng et al, 2021;Wang et al, 2021) removes the explicit event proposing process. Deng et al (2021) tackles the DVC task from a top-down perspective, in which they first generate a video-level story, then ground each sentence in the story into a video segment. Wang et al (2021) considers the DVC task as a set prediction problem, and applies two parallel prediction heads for event localization and captioning.…”

Section: Multimodal Transformermentioning

confidence: 99%

See 1 more Smart Citation

End-to-end Dense Video Captioning as Sequence Generation

Zhu¹,

Pang²,

Thapliyal³

et al. 2022

Preprint

View full text Add to dashboard Cite

Dense video captioning aims to identify the events of interest in an input video, and generate descriptive captions for each event. Previous approaches usually follow a two-stage generative process, which first proposes a segment for each event, then renders a caption for each identified segment. Recent advances in large-scale sequence generation pretraining have seen great success in unifying task formulation for a great variety of tasks, but so far, more complex tasks such as dense video captioning are not able to fully utilize this powerful paradigm. In this work, we show how to model the two subtasks of dense video captioning jointly as one sequence generation task, and simultaneously predict the events and the corresponding descriptions. Experiments on YouCook2 and ViTT show encouraging results and indicate the feasibility of training complex tasks such as end-to-end dense video captioning integrated into large-scale pretrained models.

show abstract

Section: Multimodal Transformermentioning

confidence: 99%

Section: Multimodal Transformermentioning

confidence: 99%

End-to-end Dense Video Captioning as Sequence Generation

Zhu¹,

Pang²,

Thapliyal³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Typical DVC models [30,[43][44][45]60,74,81,82] follow a two-stage, bottomup paradigm: first parse a video into several temporal events and then decode a description from each detected event. As the problem of event detection is ill-defined [10], some alterative solutions either adopt a single-stage strategy to simultaneously predict events and descriptions [35,71], or turn to a top-down regime: first generate paragraphs, and then ground each description to a video segment [10,37]. A few other methods [23,32,50] focus purely on generating better paragraph captions from a provided list of events.…”

Section: Related Workmentioning

confidence: 99%

Visual Abductive Reasoning

Chen¹,

Yang²,

Zhou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Abductive reasoning seeks the likeliest possible explanation for partial observations. Although abduction is frequently employed in human daily reasoning, it is rarely explored in computer vision literature. In this paper, we propose a new task and dataset, Visual Abductive Reasoning (VAR), for examining abductive reasoning ability of machine intelligence in everyday visual situations. Given an incomplete set of visual events, AI systems are required to not only describe what is observed, but also infer the hypothesis that can best explain the visual premise. Based on our large-scale VAR dataset, we devise a strong baseline model, REASONER (causal-and-cascaded reasoning Transformer). First, to capture the causal structure of the observations, a contextualized directional position embedding strategy is adopted in the encoder, that yields discriminative representations for the premise and hypothesis. Then, multiple decoders are cascaded to generate and progressively refine the premise and hypothesis sentences. The prediction scores of the sentences are used to guide cross-sentence information flow in the cascaded reasoning procedure. Our VAR benchmarking results show that REASONER surpasses many famous video-language models, while still being far behind human performance. This work is expected to foster future efforts in the reasoning-beyond-observation paradigm.* Part of this work was done when Chen Liang was an intern at Baidu.

show abstract

“…Some tasks use joint image and language inputs, such as captioning or VQA [57,7,8,1,18,20], while some use image and language learning to joint space which enables cross-modal retrieval [25,51,60]. Additional modalities are also often incorporated in such models, for example, video, audio, or additional text such as transcriptions or video captions [70,55,12,22,35].…”

Section: Related Workmentioning

confidence: 99%

Answer-Me: Multi-Task Open-Vocabulary Visual Question Answering

Piergiovanni¹,

Li²,

Kuo³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present Answer-Me, a task-aware multi-task framework which unifies a variety of question answering tasks, such as, visual question answering, visual entailment, visual reasoning. In contrast to previous works using contrastive or generative captioning training, we propose a novel and simple recipe to pre-train a vision-language joint model, which is multi-task as well. The pre-training uses only noisy image captioning data, and is formulated to use the entire architecture end-toend with both a strong language encoder and decoder. Our results show state-of-the-art performance, zero-shot generalization, robustness to forgetting, and competitive single-task results across a variety of question answering tasks. Our multi-task mixture training learns from tasks of various question intents and thus generalizes better, including on zeroshot vision-language tasks. We conduct experiments in the challenging multi-task and open-vocabulary settings and across a variety of datasets and tasks, such as VQA2.0, SNLI-VE, NLVR2, GQA, VizWiz. We observe that the proposed approach is able to generalize to unseen tasks and that more diverse mixtures lead to higher accuracy in both known and novel tasks.

show abstract

Sketch, Ground, and Refine: Top-Down Dense Video Captioning

Cited by 57 publications

References 27 publications

End-to-end Dense Video Captioning as Sequence Generation

End-to-end Dense Video Captioning as Sequence Generation

Visual Abductive Reasoning

Answer-Me: Multi-Task Open-Vocabulary Visual Question Answering

Contact Info

Product

Resources

About