BERT Representations for Video Question Answering

Yang, Zekun; García, Noa; Chu, Chenhui; Otani, Mayu; Nakashima, Yuta; Takemura, Haruo

doi:10.1109/wacv45572.2020.9093596

Cited by 92 publications

(49 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Anderson et al [1] exploits object-level attention with bottom-up attention, then associates the output sequences with salient image regions via top-down mechanism. More recently, self-attention networks introduced by Transformers [27] are widely adapted in both language and vision tasks [9,22,26,39,40]. Guo et al [11] normalizes the self-attention module in the transformer to solve the internal covariate shift.…”

Section: Related Workmentioning

confidence: 99%

Group-based Distinctive Image Captioning with Memory Attention

Wang

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Describing images using natural language is widely known as image captioning, which has made consistent progress due to the development of computer vision and natural language generation techniques. Though conventional captioning models achieve high accuracy based on popular metrics, i.e., BLEU, CIDEr, and SPICE, the ability of captions to distinguish the target image from other similar images is under-explored. To generate distinctive captions, a few pioneers employ contrastive learning or re-weighted the ground-truth captions, which focuses on one single input image. However, the relationships between objects in a similar image group (e.g., items or properties within the same album or fine-grained events) are neglected. In this paper, we improve the distinctiveness of image captions using a Group-based Distinctive Captioning Model (GdisCap), which compares each image with other images in one similar group and highlights the uniqueness of each image. In particular, we propose a group-based memory attention (GMA) module, which stores object features that are unique among the image group (i.e., with low similarity to objects in other images). These unique object features are highlighted when generating captions, resulting in more distinctive captions. Furthermore, the distinctive words in the ground-truth captions are selected to supervise the language decoder and GMA. Finally, we propose a new evaluation metric, distinctive word rate (DisWordRate) to measure the distinctiveness of captions. Quantitative results indicate that the proposed method significantly improves the distinctiveness of several baseline models, and achieves the state-of-the-art performance on both accuracy and distinctiveness. Results of a user study agree with the quantitative evaluation and demonstrate the rationality of the new metric DisWordRate. CCS CONCEPTS• Computing methodologies → Computer vision.

show abstract

Section: Related Workmentioning

confidence: 99%

Group-based Distinctive Image Captioning with Memory Attention

Wang

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…For example, LSTMs [12] are used to embed the temporal visual and textual features in [15], and attention mechanisms [24], which allows to focus only on specific parts of the input, bring significant improvements in [6,30,34,35]. Other work [32,33] applies Transformers [5] to capture the information from videos. For better fusion of independent visual and textual sources, Hirota et al [10,11] propose to use the textual representations to understand the visual sources.…”

Section: Related Workmentioning

confidence: 99%

Transferring Domain-Agnostic Knowledge in Video Question Answering

Wu¹,

García²,

Otani³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Video question answering (VideoQA) is designed to answer a given question based on a relevant video clip. The current available large-scale datasets have made it possible to formulate VideoQA as the joint understanding of visual and language information. However, this training procedure is costly and still less competent with human performance. In this paper, we investigate a transfer learning method by the introduction of domain-agnostic knowledge and domain-specific knowledge. First, we develop a novel transfer learning framework, which finetunes the pre-trained model by applying domainagnostic knowledge as the medium. Second, we construct a new VideoQA dataset with 21, 412 human-generated question-answer samples for comparable transfer of knowledge. Our experiments show that: (i) domain-agnostic knowledge is transferable and (ii) our proposed transfer learning framework can boost VideoQA performance effectively.

show abstract

“…It integrates the information in multiple modalities and therefore is expected to perform better prediction than the case using any unimodal information [1]. Nowadays it has been applied in a broad range of applications, such as multimedia event detection [2,3], sentiment analysis [1,4], cross-modal translation [5][6][7], Visual Question Answering (VQA) [8,9], etc.…”

Section: Introductionmentioning

confidence: 99%

A Multi-Modal Fusion Method Based on Higher-Order Orthogonal Iteration Decomposition

Fen

Chen

Tan

et al. 2021

Entropy

View full text Add to dashboard Cite

Multi-modal fusion can achieve better predictions through the amalgamation of information from different modalities. To improve the performance of accuracy, a method based on Higher-order Orthogonal Iteration Decomposition and Projection (HOIDP) is proposed, in the fusion process, higher-order orthogonal iteration decomposition algorithm and factor matrix projection are used to remove redundant information duplicated inter-modal and produce fewer parameters with minimal information loss. The performance of the proposed method is verified by three different multi-modal datasets. The numerical results validate the accuracy of the performance of the proposed method having 0.4% to 4% improvement in sentiment analysis, 0.3% to 8% improvement in personality trait recognition, and 0.2% to 25% improvement in emotion recognition at three different multi-modal datasets compared with other 5 methods.

show abstract

BERT Representations for Video Question Answering

Cited by 92 publications

References 30 publications

Group-based Distinctive Image Captioning with Memory Attention

Group-based Distinctive Image Captioning with Memory Attention

Transferring Domain-Agnostic Knowledge in Video Question Answering

A Multi-Modal Fusion Method Based on Higher-Order Orthogonal Iteration Decomposition

Contact Info

Product

Resources

About