Explore Multi-Step Reasoning in Video Question Answering

Song, Xiaomeng; Shi, Yucheng; Chen, Xin; Han, Yahong

doi:10.1145/3240508.3240563

Cited by 49 publications

(36 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Earlier efforts (Andreas et al, 2016; designed questions that are expressed as elementary operation programs. More related to our work, Song et al (2018);Yi* et al (2020) extended the prior work to the video domain with questions focusing on the temporal variance of video frames.…”

Section: Related Workmentioning

confidence: 99%

“…In (Antol et al, 2015), Visual7W (Zhu et al, 2016 TGIF-QA (Jang et al, 2017), TV-QA (Lei et al, 2018) IQA (Gordon et al, 2018), EQA (Wijmans et al, 2019) Image/video grounded dialogues, navigation dialogues VisDial ), GuessWhat (De Vries et al, 2017 AVSD , CVDN (Thomason et al, 2019) Synthetic image/video QA SHAPE (Andreas et al, 2016), CLEVR SVQA (Song et al, 2018), CLEVRER (Yi* et al, 2020) Synthetic dialogues bAbI (Bordes et al, 2017) MNIST Dialog (Seo et al, 2017), CLEVR-Dialog (Kottur et al, 2019)…”

Section: Dvd Question Types Sub-types and Examplesmentioning

confidence: 99%

See 1 more Smart Citation

DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue

Lê

Sankar

Moon

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

A video-grounded dialogue system is required to understand both dialogue, which contains semantic dependencies from turn to turn, and video, which contains visual cues of spatial and temporal scene variations. Building such dialogue systems is a challenging problem, involving various reasoning types on both visual and language inputs. Existing benchmarks do not have enough annotations to thoroughly analyze dialogue systems and understand their capabilities and limitations in isolation. These benchmarks are also not explicitly designed to minimise biases that models can exploit without actual reasoning. To address these limitations, in this paper, we present DVD, a Diagnostic Dataset for Videogrounded Dialogues. The dataset is designed to contain minimal biases and has detailed annotations for the different types of reasoning over the spatio-temporal space of video. Dialogues are synthesized over multiple question turns, each of which is injected with a set of cross-turn semantic relationships. We use DVD to analyze existing approaches, providing interesting insights into their abilities and limitations. In total, DVD is built from 11k CATER synthetic videos and contains 10 instances of 10-round dialogues for each video, resulting in more than 100k dialogues and 1M question-answer pairs. Our code and dataset are publicly available 1 .

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Dvd Question Types Sub-types and Examplesmentioning

confidence: 99%

DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue

Lê

Sankar

Moon

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…We evaluate our proposed architecture on the three recent benchmarks, namely, MSVD-QA, MSRVTT-QA [30], [31] and SVQA [26].…”

Section: Experiments a Datasetsmentioning

confidence: 99%

“…We train our models with Adam optimizer with an initial learning rate of 10 −4 and a batch size of 64. To be compatible with related works [20], [26], [19], we use accuracy as evaluation metric for all tasks.…”

Section: Modelmentioning

confidence: 99%

“…The system is evaluated on three major Video QA datasets: MSVD-QA [30] consisting of 50K questions with 2K videos, MSRVTT-QA [31] with 243K questions over 10K videos, and SVQA [26] with 120K questions over 12K videos of moving objects. These datasets are suitable to test the reasoning capability against complex compositional questions about spatialtemporal relationships among objects.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Object-Centric Representation Learning for Video Question Answering

Dang¹,

Le²,

Le³

et al. 2021

Preprint

View full text Add to dashboard Cite

Video question answering (Video QA) presents a powerful testbed for human-like intelligent behaviors. The task demands new capabilities to integrate video processing, language understanding, binding abstract linguistic concepts to concrete visual artifacts, and deliberative reasoning over spacetime. Neural networks offer a promising approach to reach this potential through learning from examples rather than handcrafting features and rules. However, neural networks are predominantly feature-based -they map data to unstructured vectorial representation and thus can fall into the trap of exploiting shortcuts through surface statistics instead of true systematic reasoning seen in symbolic systems. To tackle this issue, we advocate for object-centric representation as a basis for constructing spatio-temporal structures from videos, essentially bridging the semantic gap between low-level pattern recognition and high-level symbolic algebra. To this end, we propose a new query-guided representation framework to turn a video into an evolving relational graph of objects, whose features and interactions are dynamically and conditionally inferred. The object lives are then summarized into résumés, lending naturally for deliberative relational reasoning that produces an answer to the query. The framework is evaluated on major Video QA datasets, demonstrating clear benefits of the object-centric approach to video reasoning.

show abstract

Describing Unseen Videos via Multi-modal Cooperative Dialog Agents

Zhu

Yang

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Explore Multi-Step Reasoning in Video Question Answering

Cited by 49 publications

References 25 publications

DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue

DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue

Object-Centric Representation Learning for Video Question Answering

Describing Unseen Videos via Multi-modal Cooperative Dialog Agents

Contact Info

Product

Resources

About