Multi-interaction Network with Object Relation for Video Question Answering

Jin, Weike; Zhao, Zhou; Gu, Mao; Yu, Jun; Xiao, Jun; Zhuang, Yueting

doi:10.1145/3343031.3351065

Cited by 59 publications

(19 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Video-Text Matching Video-text matching has been widely studied in various tasks, such as video retrieval (Liu et al, 2019;, moment localization with natural language (Zhang et al, 2019(Zhang et al, , 2020 and video question and answering (Xu et al, 2017;Jin et al, 2019b). It aims to learn video-semantic representation in a joint embedding space.…”

Section: Related Workmentioning

confidence: 99%

Video-aided Unsupervised Grammar Induction

Zhang

Song

Jin

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

We investigate video-aided grammar induction, which learns a constituency parser from both unlabeled text and its corresponding video. Existing methods of multi-modal grammar induction focus on learning syntactic grammars from text-image pairs, with promising results showing that the information from static images is useful in induction. However, videos provide even richer information, including not only static objects but also actions and state changes useful for inducing verb phrases. In this paper, we explore rich features (e.g. action, object, scene, audio, face, OCR and speech) from videos, taking the recent Compound PCFG model (Kim et al., 2019) as the baseline. We further propose a Multi-Modal Compound PCFG model (MMC-PCFG) to effectively aggregate these rich features from different modalities. Our proposed MMC-PCFG is trained end-to-end and outperforms each individual modality and previous state-of-the-art systems on three benchmarks, i.e. DiDeMo, YouCook2 and MSRVTT, confirming the effectiveness of leveraging video information for unsupervised grammar induction.

show abstract

Section: Related Workmentioning

confidence: 99%

Video-aided Unsupervised Grammar Induction

Zhang

Song

Jin

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

show abstract

“…To answer such semantic-complicated questions that are based on fine-grained comprehension of video content, relational reasoning-based methods [4]- [6], [14]- [16], [20] are proposed. More specifically, Jin et al [4] propose a multimodal and multi-level interaction network to capture relations between objects. Jiang et al [14] develop a heterogenous graph alignment network to integrate the relations of both inter-and intra-modality for cross-modal reasoning.…”

Section: A Video Question Answeringmentioning

confidence: 99%

“…1) Comparisons on MSRVTT-QA: For MSRVTT-QA, we compare the proposed LiVLR with recent methods, including Park et al [21], DualVGR [15], HGA [14], MASN [6], HCR [16], MiNOR [4], HME [10], GRA [1], Co-mem [3], ST-VQA [2], VQA-T [13], CoMVT [68], ClipBERT [12], and SSML [67]. It is worth noting that VQA-T, CoMVT, ClipBERT, and SSML (x) adopt large-scale video-language pretraining to enhance the downstream VideoQA task.…”

Section: B Comparisons With State-of-the-artsmentioning

confidence: 99%

“…and Penny in one frame of the video. To this end, the relational reasoning-based VideoQA methods [4]- [6] are proposed to model relationships between visual objects. In addition to the aforementioned multi-grained visual content, there are some videos contain linguistic content, such as subtitles [7], [8], knowledge [9], and descriptions [1].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

LiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering

Jiang¹,

Liu²,

Zheng³

2021

Preprint

View full text Add to dashboard Cite

Video Question Answering (VideoQA), aiming to correctly answer the given question based on understanding multi-modal video content, is challenging due to the rich video content. From the perspective of video understanding, a good VideoQA framework needs to understand the video content at different semantic levels and flexibly integrate the diverse video content to distill question-related content. To this end, we propose a Lightweight Visual-Linguistic Reasoning framework named LiVLR. Specifically, LiVLR first utilizes the graph-based Visual and Linguistic Encoders to obtain multi-grained visual and linguistic representations. Subsequently, the obtained representations are integrated with the devised Diversity-aware Visual-Linguistic Reasoning module (DaVL). The DaVL considers the difference between the different types of representations and can flexibly adjust the importance of different types of representations when generating the question-related joint representation, which is an effective and general representation integration method. The proposed LiVLR is lightweight and shows its performance advantage on two VideoQA benchmarks, MRSVTT-QA and KnowIT VQA. Extensive ablation studies demonstrate the effectiveness of LiVLR key components.

show abstract

“…Some researchers have presented to capture fine-grained appearance-question interactions. Jin et al [13] introduced object-aware temporal attention that learns object-question interactions. Huang etal [10] also utilize frame and object features to enhance coattention between the appearance and question.…”

Section: Related Workmentioning

confidence: 99%

Bridge to Answer: Structure-aware Graph Interaction Network for Video Question Answering

Park¹,

Lee²,

Sohn³

2021

Preprint

View full text Add to dashboard Cite

This paper presents a novel method, termed Bridge to Answer, to infer correct answers for questions about a given video by leveraging adequate graph interactions of heterogeneous crossmodal graphs. To realize this, we learn question conditioned visual graphs by exploiting the relation between video and question to enable each visual node using question-to-visual interactions to encompass both visual and linguistic cues. In addition, we propose bridged visualto-visual interactions to incorporate two complementary visual information on appearance and motion by placing the question graph as an intermediate bridge. This bridged architecture allows reliable message passing through compositional semantics of the question to generate an appropriate answer. As a result, our method can learn the question conditioned visual representations attributed to appearance and motion that show powerful capability for video question answering. Extensive experiments prove that the proposed method provides effective and superior performance than state-of-the-art methods on several benchmarks.

show abstract

Multi-interaction Network with Object Relation for Video Question Answering

Cited by 59 publications

References 29 publications

Video-aided Unsupervised Grammar Induction

Video-aided Unsupervised Grammar Induction

LiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering

Bridge to Answer: Structure-aware Graph Interaction Network for Video Question Answering

Contact Info

Product

Resources

About