DeepStory: Video Story QA by Deep Embedded Memory Networks

Kim, Kyung-Min; Heo, Min-Oh; Choi, Seong-Ho; Zhang, Byoung-Tak

doi:10.48550/arxiv.1707.00836

Cited by 22 publications

(19 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compare FVTA with recent results on MovieQA dataset, including End-to-End Memory Network (MemN2N) [23], Deep Embedded Memory Network (DEMN) [10], and Read-Write Memory Network (RWMN) [15]. Table 4 shows the detailed comparison of MovieQA results using both videos and subtitles.…”

Section: Resultsmentioning

confidence: 99%

“…A recent direction is on the question answering based on videos, which is more relevant to this work. A number of research studies have been carried on MovieQA [22,10,15], with movie clips, scripts, and descriptions. Because it is expensive to 1 Code and models are released at https://memexqa.cs.cmu.…”

Section: Related Workmentioning

confidence: 99%

“…This also leaves significant rooms of improvement with visual information. [22] 0.219 -MemN2N [22] 0.342 -DEMN [10] -0.300 Soft Attention 0.321 -MCB [5] 0.362 -TGIF Temporal [7] 0.371 -RWMN [15] 0.387 0.363 FVTA 0.410 0.373 clips from the same movie and the corresponding subtitles.…”

Section: Ablation Studymentioning

confidence: 99%

See 2 more Smart Citations

Focal Visual-Text Attention for Visual Question Answering

Liang

Jiang

Cao³

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

View full text Add to dashboard Cite

Recent insights on language and vision with neural networks have been successfully applied to simple singleimage visual question answering. However, to tackle reallife question answering problems on multimedia collections such as personal photos, we have to look at whole collections with sequences of photos or videos. When answering questions from a large collection, a natural problem is to identify snippets to support the answer. In this paper, we describe a novel neural network called Focal Visual-Text Attention network (FVTA) for collective reasoning in visual question answering, where both visual and text sequence information such as images and text metadata are presented. FVTA introduces an end-to-end approach that makes use of a hierarchical process to dynamically determine what media and what time to focus on in the sequential data to answer the question. FVTA can not only answer the questions well but also provides the justifications which the system results are based upon to get the answers. FVTA achieves state-of-the-art performance on the MemexQA dataset and competitive results on the MovieQA dataset.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Focal Visual-Text Attention for Visual Question Answering

Liang

Jiang

Cao³

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

View full text Add to dashboard Cite

show abstract

“…[3]. Other video VQA datasets include social questions in addition to other more factual questions, such as the TVQA dataset that uses videos from six wellknown TV shows [16]; PororoQA, that uses videos from a children's animated television show [17]; and MovieQA that contains questions about movies [18].…”

Section: B Social Vqa For Ai Agentsmentioning

confidence: 99%

Characterizing Datasets for Social Visual Question Answering, and the New TinySocial Dataset

Chen¹,

Li²,

Rashedi³

et al. 2020

2020 Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob)

View full text Add to dashboard Cite

Modern social intelligence includes the ability to watch videos and answer questions about social and theoryof-mind-related content, e.g., for a scene in Harry Potter, "Is the father really upset about the boys flying the car?" Social visual question answering (social VQA) is emerging as a valuable methodology for studying social reasoning in both humans (e.g., children with autism) and AI agents. However, this problem space spans enormous variations in both videos and questions. We discuss methods for creating and characterizing social VQA datasets, including 1) crowdsourcing versus in-house authoring, including sample comparisons of two new datasets that we created (TinySocial-Crowd and TinySocial-InHouse) and the previously existing Social-IQ dataset; 2) a new rubric for characterizing the difficulty and content of a given video; and 3) a new rubric for characterizing question types. We close by describing how having well-characterized social VQA datasets will enhance the explainability of AI agents and can also inform assessments and educational interventions for people.

show abstract

“…Hence, to study the task of multimodal moment retrieval with both video and subtitle text contexts, we propose a new dataset -TV show Retrieval (TVR). Inspired by recent works [30,15,18] which relied on Movie/Cartoon/TV shows for building multimodal datasets, we select TV shows as our data collection resource as they typically involve rich social interactions between actors, involving both activities and dialogues. During data collection, we present annotators with videos and associated subtitles to encourage them to write multimodal queries.…”

Section: Introductionmentioning

confidence: 99%

TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval

Lei

Berg

et al. 2020

Preprint

View full text Add to dashboard Cite

We introduce a new multimodal retrieval task -TV show Retrieval (TVR), in which a short video moment has to be localized from a large video (with subtitle) corpus, given a natural language query. Different from previous moment retrieval tasks dealing with videos only, TVR requires the system to understand both the video and the associated subtitle text, making it a more realistic task. To support the study of this new task, we have collected a large-scale, high-quality dataset consisting of 108,965 queries on 21,793 videos from 6 TV shows of diverse genres, where each query is associated with a tight temporal alignment. Strict qualification and post-annotation verification tests are applied to ensure the quality of the collected data. We present several baselines and a novel Cross-modal Moment Localization (XML) modular network for this new dataset and task. The proposed XML model surpasses all presented baselines by a large margin and with better efficiency, providing a strong starting point for future work. Extensive analysis experiments also show that incorporating both video and subtitle modules yields better performance than either alone. Lastly, we have also collected additional descriptions for each annotated moment in TVR to form a new multimodal captioning dataset with 262K captions, named the TV show Caption dataset (TVC). Here models need to jointly use the video and subtitle to generate a caption description. Both datasets are publicly available at https://tvr.cs.unc.edu.

show abstract

DeepStory: Video Story QA by Deep Embedded Memory Networks

Cited by 22 publications

References 9 publications

Focal Visual-Text Attention for Visual Question Answering

Focal Visual-Text Attention for Visual Question Answering

Characterizing Datasets for Social Visual Question Answering, and the New TinySocial Dataset

TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval

Contact Info

Product

Resources

About