Video Question Answering via Hierarchical Dual-Level Attention Network Learning

Zhao, Zhou; Lin, Jinghao; Jiang, Xinghua; He, Xiaofei; Zhuang, Yueting

doi:10.1145/3123266.3123364

Cited by 35 publications

(5 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One of the first datasets for VQA is YouTube2Text [48] which comprises of 1987 videos and 122,708 descriptions of those in natural language. Many existing models [5,9,13,21] address the task of VQA with the help of YouTube2Text [48]. The LSMDC 2016 description dataset [47] enables multi-sentence description of the videos which is a unique property from the previous approaches of datasets.…”

Section: Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

Recent Advances in Video Question Answering: A Review of Datasets and Methods

Patel,

Parikh,

Shastri

2021

Preprint

View full text Add to dashboard Cite

Video Question Answering (VQA) is a recent emerging challenging task in the field of Computer Vision. Several visual information retrieval techniques like Video Captioning/Description and Video-guided Machine Translation have preceded the task of VQA. VQA helps to retrieve temporal and spatial information from the video scenes and interpret it. In this survey, we review a number of methods and datasets for the task of VQA. To the best of our knowledge, no previous survey has been conducted for the VQA task.

show abstract

Section: Datasetsmentioning

confidence: 99%

“…In [9], authors study the problem of VQA from the viewpoint of Hierarchical Dual-Level Attention Network (DLAN) learning. On the basis of frame-level and segment-level feature representations, object appearance and movement information from the video is obtained.…”

Section: Dlan (Zao Et Al 2017)mentioning

confidence: 99%

Recent Advances in Video Question Answering: A Review of Datasets and Methods

Patel,

Parikh,

Shastri

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Motivated by the success of attention mechanism [53]- [55], a self-attentive pooling is adopted to explicitly capture the varying importance of each sequential unit by assigning it an attention weight. For each modality, the self-attentive pooling takes the whole output sequence…”

Section: ) Self-attentive Sequential Feature Learningmentioning

confidence: 99%

Sentiment Enhanced Multi-Modal Hashtag Recommendation for Micro-Videos

2020

View full text Add to dashboard Cite

Recommending hashtags for micro-videos is a challenging task due to the following two reasons: 1) micro-video is a unity of multi-modalities, including the visual, acoustic, and textual modalities. Therefore, how to effectively extract features from multi-modalities and utilize them to express the microvideo is of great significance; 2) micro-videos usually include moods and feelings, which may provide crucial cues for recommending proper hashtags. However, most of the existing works have not considered the sentiment of media data for hashtag recommendation. In this paper, the senTiment enhanced multi-mOdal Attentive haShtag recommendaTion (TOAST) model is proposed for micro-video hashtag recommendation. Different from previous hashtag recommendation models, which merely consider content features, sentiment features of modalities are further incorporated in TOAST to improve the recommendation performance of the sentiment hashtags (e.g., #funny, #sad). Specifically, the multi-modal content features and the multi-modal sentiment features are modeled by a content common space learning branch based on self-attention and a sentiment common space learning branch, respectively. Furthermore, the varying importance of the multimodal sentiment and content features are dynamically captured via an attention neural network according to their consistency with the hashtag semantic embedding by an attention neural network. Extensive experiments on a real-world dataset have demonstrated the effectiveness of the proposed method compared with the baseline methods. Meanwhile, the findings from the experiments may provide new insight for future developments of micro-video hashtag recommendation. INDEX TERMS Hashtag recommendation, micro-videos, multiple modalities, self-attention mechanism, sentiment features. I. INTRODUCTION Nowadays, watching micro-videos for leisure and entertainment has gained tremendous user enthusiasm. Taking China as an example, the number of micro-video users has risen from 501 million in 2018 to 627 million in 2019, and is predicted to growth to 722 million in 2020 according to the reports in iiMedia. 1 And micro-video platforms and apps, such as Vine, 2 Snapchat, 3 Kuaishou, 4 and Douyin, 5 etc., have also received unprecedented growth in recent years. How to facilitate users to quickly and accurately find their desired The associate editor coordinating the review of this manuscript and approving it for publication was Yin Zhang .

show abstract

“…), which aims to answer a natural language question according to a video clip, is an important task in multimedia understanding. Modern Video QA systems [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13] require sufficient video-question-answer triples to train. As shown in Figure 1, many Video QA datasets are labeled with caption question generation (CapQG), where question-answer pairs are created using a text question generation system according to captions of a video.…”

Section: Introductionmentioning

confidence: 99%

End-to-End Video Question-Answer Generation with Generator-Pretester Network

Chang

Shen

et al. 2021

Preprint

View full text Add to dashboard Cite

We study a novel task, Video Question-Answer Generation (VQAG), for challenging Video Question Answering (Video QA) task in multimedia. Due to expensive data annotation costs, many widely used, large-scale Video QA datasets such as Video-QA, MSVD-QA and MSRVTT-QA are automatically annotated using Caption Question Generation (CapQG) which inputs captions instead of the video itself. As captions neither fully represent a video, nor are they always practically available, it is crucial to generate question-answer pairs based on a video via Video Question-Answer Generation (VQAG). Existing videoto-text (V2T) approaches, despite taking a video as the input, only generate a question alone. In this work, we propose a novel model Generator-Pretester Network that focuses on two components: (1) The Joint Question-Answer Generator (JQAG) which generates a question with its corresponding answer to allow Video Question "Answering" training. (2) The Pretester (PT) verifies a generated question by trying to answer it and checks the pretested answer with both the model's proposed answer and the ground truth answer. We evaluate our system with the only two available large-scale human-annotated Video QA datasets and achieves state-of-the-art question generation performances. Furthermore, using our generated QA pairs only on the Video QA task, we can surpass some supervised baselines. As a pre-training strategy, we outperform both CapQG and transfer learning approaches when employing semi-supervised (20%) or fully supervised learning with annotated data. These experimental results suggest the novel perspectives for Video QA training.

show abstract

Video Question Answering via Hierarchical Dual-Level Attention Network Learning

Cited by 35 publications

References 38 publications

Recent Advances in Video Question Answering: A Review of Datasets and Methods

Recent Advances in Video Question Answering: A Review of Datasets and Methods

Sentiment Enhanced Multi-Modal Hashtag Recommendation for Micro-Videos

End-to-End Video Question-Answer Generation with Generator-Pretester Network

Contact Info

Product

Resources

About