DramaQA: Character-Centered Video Story Understanding with Hierarchical QA

Choi, Seong-Ho; On, Kyoung-Woon; Heo, Yu-Jung; Seo, Ahjeong; Jang, Youwon; Lee, Seung-Chan; Lee, Min Su; Zhang, Byoung-Tak

doi:10.5626/ktcp.2021.27.1.7

Cited by 11 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Three QA tasks are used for the verification of the proposed model: RACE [36], QuAC [37], and DramaQA [9]. RACE is a data set for the multiple choice QA where a question is composed of a questionary sentence, an associated passage, and a set of candidate answers.…”

Section: Experimental Settingmentioning

confidence: 99%

“…In this section, we solve the question answering with a predicted difficulty level to verify that the performance of question answering is improved with the difficulty level. We choose the multi-level context matching model [9] as a question answering model, since is currently the state-of-the-art model for the DramaQA QA task. The multi-level context matching model is designed to understand the multimodal story of a drama.…”

Section: Performance Of Question Answering With Difficulty Levelmentioning

confidence: 99%

“…Therefore, the QA models attacking the tasks do not recognize the difficulty of each question even though the difficulty is important information to answer the questions [8]. As a result, a difficulty level is tagged in new QA tasks such as DramaQA [9] in conjunction with Piaget's theory [10].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Question Difficulty Estimation Based on Attention Model for Question Answering

2021

View full text Add to dashboard Cite

This paper addresses a question difficulty estimation of which goal is to estimate the difficulty level of a given question in question-answering (QA) tasks. Since a question in the tasks is composed of a questionary sentence and a set of information components such as a description and candidate answers, it is important to model the relationship among the information components to estimate the difficulty level of the question. However, existing approaches to this task modeled a simple relationship such as a relationship between a questionary sentence and a description, but such simple relationships are insufficient to predict the difficulty level accurately. Therefore, this paper proposes an attention-based model to consider the complicated relationship among the information components. The proposed model first represents bi-directional relationships between a questionary sentence and each information component using a dual multi-head co-attention, since the questionary sentence is a key factor in the QA questions and it affects and is affected by information components. Then, the proposed model considers inter-information relationship over the bi-directional representations through a self-attention model. The inter-information relationship helps predict the difficulty of the questions accurately which require reasoning over multiple kinds of information components. The experimental results from three well-known and real-world QA data sets prove that the proposed model outperforms the previous state-of-the-art and pre-trained language model baselines. It is also shown that the proposed model is robust against the increase of the number of information components.

show abstract

Section: Experimental Settingmentioning

confidence: 99%

Section: Performance Of Question Answering With Difficulty Levelmentioning

confidence: 99%

See 1 more Smart Citation

Question Difficulty Estimation Based on Attention Model for Question Answering

2021

View full text Add to dashboard Cite

show abstract

“…Thanks to this, visionlanguage tasks such as image captioning (Xu et al, 2015), visual question answering (VQA) (Antol et al, 2015;Goyal et al, 2017), and visual commonsense reasoning (VCR) (Zellers et al, 2019) have been introduced to the research community, along with some benchmark datasets. In particular, video question answering (video QA) tasks (Xu et al, 2016;Jang et al, 2017;Yu et al, 2019;Choi et al, 2020) have been proposed with the goal of reasoning over higher-level visionlanguage interactions. In contrast to QA tasks based on static images, the questions presented in the video QA dataset vary from frame-level questions regarding the appearance of objects (e.g., what is the color of the hat?)…”

Section: Introductionmentioning

confidence: 99%

Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering

Seo¹,

Kang²,

Park³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

Self Cite

View full text Add to dashboard Cite

Video Question Answering is a task which requires an AI agent to answer questions grounded in video. This task entails three key challenges: (1) understand the intention of various questions, (2) capturing various elements of the input video (e.g., object, action, causality), and (3) cross-modal grounding between language and vision information. We propose Motion-Appearance Synergistic Networks (MASN), which embed two crossmodal features grounded on motion and appearance information and selectively utilize them depending on the question's intentions. MASN consists of a motion module, an appearance module, and a motion-appearance fusion module. The motion module computes the action-oriented cross-modal joint representations, while the appearance module focuses on the appearance aspect of the input video. Finally, the motion-appearance fusion module takes each output of the motion module and the appearance module as input, and performs question-guided fusion. As a result, MASN achieves new state-of-the-art performance on the TGIF-QA and MSVD-QA datasets. We also conduct qualitative analysis by visualizing the inference results of MASN. The code is available at https://github.com/ ahjeongseo/MASN-pytorch.

show abstract

“…Recently, researches on text-to-video retrieval, video captioning, and video question answering (videoQA) have been actively conducted to improve video understanding intelligence. In addition, large-scale datasets have been built and publicly available to facilitate the researches (Alamri et al 2019;Lei et al 2018Lei et al , 2020Choi et al 2021). Studies using these datasets usually apply automatic evaluation metrics to measure the performances of AI agents.…”

Section: Introductionmentioning

confidence: 99%

Toward a Human-Level Video Understanding Intelligence

Heo,

Lee,

Choi

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We aim to develop an AI agent that can watch video clips and have a conversation with human about the video story. Developing video understanding intelligence is a significantly challenging task, and evaluation methods for adequately measuring and analyzing the progress of AI agent are lacking as well. In this paper, we propose the Video Turing Test to provide effective and practical assessments of video understanding intelligence as well as human-likeness evaluation of AI agents. We define a general format and procedure of the Video Turing Test and present a case study to confirm the effectiveness and usefulness of the proposed test.

show abstract

DramaQA: Character-Centered Video Story Understanding with Hierarchical QA

Cited by 11 publications

References 0 publications

Question Difficulty Estimation Based on Attention Model for Question Answering

Question Difficulty Estimation Based on Attention Model for Question Answering

Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering

Toward a Human-Level Video Understanding Intelligence

Contact Info

Product

Resources

About