Compact Trilinear Interaction for Visual Question Answering

Do, Tuong; Tran, Huy Dat; Do, Thanh-Toan; Tjiputra, Erman; Tran, Quang D.

doi:10.1109/iccv.2019.00048

Cited by 62 publications

(46 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The aggregator needs to detect high-level interactions between the three streams to provide a meaningful answer, without erasing the lower-level interactions extracted in the previous steps. We design the aggregator by applying the Compact Trilinear Interaction (CTI) (Do et al, 2019) to question, answer, and image features and generate a vector to jointly represent the three features.…”

Section: Concept-vision-language Embedding Modulementioning

confidence: 99%

See 1 more Smart Citation

ConceptBert: Concept-Aware Representation for Visual Question Answering

Gardères¹,

Ziaeefard²,

Abeloos³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities. Current works in VQA focus on questions which are answerable by direct analysis of the question and image alone. We present a concept-aware algorithm, ConceptBert, for questions which require common sense, or basic factual knowledge from external structured content. Given an image and a question in natural language, ConceptBert requires visual elements of the image and a Knowledge Graph (KG) to infer the correct answer. We introduce a multi-modal representation which learns a joint Concept-Vision-Language embedding. We exploit ConceptNet KG for encoding the common sense knowledge and evaluate our methodology on the Outside Knowledge-VQA (OK-VQA

show abstract

Section: Concept-vision-language Embedding Modulementioning

confidence: 99%

“…However, it leads to an increase in the computational cost. We set R = 32 in Equation 5, the same value as in the CTI (Do et al, 2019) for the slicing parameter. optimizer with an initial learning rate of 4e-5.…”

Section: Concept-vision-language Embeddingmentioning

confidence: 99%

ConceptBert: Concept-Aware Representation for Visual Question Answering

Gardères¹,

Ziaeefard²,

Abeloos³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

show abstract

“…Our GATs include 16 attention heads. In the fusion model, R = 32 as suggested in (Do et al, 2019) and d z = 512 since it leads to the best results in our model.…”

Section: Methodsmentioning

confidence: 86%

“…Popular fusion methods such as BAN (Kim et al, 2018) or MUTAN (Ben-younes et al, 2017) are not suitable for our work since we have three types of features to fuse. Therefore, we design a fusion method by applying the Compact Trilinear Interaction (CTI) (Do et al, 2019) to the question embeddings, scene graph visual features, and concept features and generate a vector to jointly represent the three features.…”

Section: Multimodal Fusionmentioning

confidence: 99%

Towards Knowledge-Augmented Visual Question Answering

Ziaeefard

Lécué

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Visual Question Answering (VQA) remains algorithmically challenging while it is effortless for humans. Humans combine visual observations with general and commonsense knowledge to answer a question about a given image. In this paper, we address the problem of incorporating general knowledge into VQA models while leveraging the visual information. We propose a model that captures the interactions between objects in a visual scene and entities in an external knowledge source. Our model is a graph-based approach that combines scene graphs with concept graphs, which learns a question-adaptive graph representation of related knowledge instances. We use Graph Attention Networks to set higher importance to key knowledge instances that are mostly relevant to each question. We exploit ConceptNet as the source of general knowledge and evaluate the performance of our model on the challenging OK-VQA dataset. Our code will be available at https:

show abstract

“…As encoderdecoder based framework is widely used for sequence machine learning, here we focus on the discussion of multi-modal fusion in the encoder stage or the decoder stage. Generally, fusing features in encoder can acquire better performance than which in decoder [38,39], as information from different modalities can interact earlier. However, we usually lack of optimal mapping between different modalities, which makes the encoder fusion challenging.…”

Section: Multi-modal Machine Learningmentioning

confidence: 99%

Multi-modal Attention Network for Handwritten Mathematical Expression Recognition

Wang

Zhang

et al. 2019

2019 International Conference on Document Analysis and Recognition (ICDAR)

View full text Add to dashboard Cite

In this paper, we propose a novel stroke constrained attention network (SCAN) which treats stroke as the basic unit for encoder-decoder based online handwritten mathematical expression recognition (HMER). Unlike previous methods which use trace points or image pixels as basic units, SCAN makes full use of stroke-level information for better alignment and representation. The proposed SCAN can be adopted in both single-modal (online or offline) and multi-modal HMER. For single-modal HMER, SCAN first employs a CNN-GRU encoder to extract point-level features from input traces in online mode and employs a CNN encoder to extract pixel-level features from input images in offline mode, then use stroke constrained information to convert them into online and offline stroke-level features. Using stroke-level features can explicitly group points or pixels belonging to the same stroke, therefore reduces the difficulty of symbol segmentation and recognition via the decoder with attention mechanism. For multi-modal HMER, other than fusing multi-modal information in decoder, SCAN can also fuse multi-modal information in encoder by utilizing the stroke based alignments between online and offline modalities. The encoder fusion is a better way for combining multi-modal information as it implements the information interaction one step before the decoder fusion so that the advantages of multiple modalities can be exploited earlier and more adequately when training the encoder-decoder model. Evaluated on a benchmark published by CROHME competition, the proposed SCAN achieves the state-of-the-art performance.

show abstract

Compact Trilinear Interaction for Visual Question Answering

Cited by 62 publications

References 35 publications

ConceptBert: Concept-Aware Representation for Visual Question Answering

ConceptBert: Concept-Aware Representation for Visual Question Answering

Towards Knowledge-Augmented Visual Question Answering

Multi-modal Attention Network for Handwritten Mathematical Expression Recognition

Contact Info

Product

Resources

About