BERT based Multiple Parallel Co-attention Model for Visual Question Answering

Dias, Mário; Aloj, Hansie; Ninan, Nijo; Koshti, Dipali

doi:10.1109/iciccs53718.2022.9788253

Cited by 2 publications

(4 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…VGGNet LSTM+Q+I [1], AVWAN [2],SAN [6],Facts-VQA [70],DPP [75], QAM [76], Region-Sel [77], NMN [78], ResNet FDA [5], Bayesian [7], Dense-Sym [8], Code-Mix VQA [9], Hei-Co-atten [10], Rich-img-Region [27], MCB [29], MRN [30] , FVTA [33], MUTAN [36], Meta-VQA [77],Rich-VQA [79], QTA [80], , DCN [81], GoogleNet Neural Image QA [80], Multi-Modal QA [82] , i-Bowing [83], Smem [84] F-RCNN Code-Mixed VQA [9], CAQT [11], QLOB [12], BAN [28] , MFB [32], [85] ,explicit-know-Based [86] , Know-Base Graph [87] BERT VilBERT [13], LXMERT [14] , UNITER [15], Oscar [16], MPC [25], Semantic VLBERT [88] Source: Own elaboration. The next step in the VQA model is to extract question features.…”

Section: Methods Papermentioning

confidence: 99%

“…Most of the recent work uses transformers for question featurization. [25,26] utilized transformer BERT for language feature extraction. In [87] authors concatenated the output of four consecutive BERT layers in order to generate hierarchical features from the question.…”

Section: Methods Papermentioning

confidence: 99%

“…VilBERT [13],LXMERT [14], UNITER [15], Oscar [16], Coarse to fine reasoning [23], MPC [25], Hie-Alternation coattention [26], Rich Image region VQA [27], KRISP [50] Source: Own elaboration.…”

Section: Grumentioning

confidence: 99%

See 2 more Smart Citations

A Comprehensive Review and Open Challenges on Visual Question Answering Models

Shaik,

Koshti,

Gupta

et al. 2023

AiBi Revista de Investigación, Administración e Ingeniería

View full text Add to dashboard Cite

Users are now able to actively interact with images and pose different questions based on images, thanks to recent developments in artificial intelligence. In turn, a response in a natural language answer is expected. The study discusses a variety of datasets that can be used to examine applications for visual question-answering (VQA), as well as their advantages and disadvantages. Four different forms of VQA models—simple joint embedding-based models, attention-based models, knowledge-incorporated models, and domain-specific VQA models—are in-depth examined in this article. We also critically assess the drawbacks and future possibilities of all current state-of-the-art (SoTa), end-to-end VQA models. Finally, we present the directions and guidelines for further development of the VQA models.

show abstract

Section: Methods Papermentioning

confidence: 99%

Section: Methods Papermentioning

confidence: 99%

See 1 more Smart Citation

A Comprehensive Review and Open Challenges on Visual Question Answering Models

Shaik,

Koshti,

Gupta

et al. 2023

AiBi Revista de Investigación, Administración e Ingeniería

View full text Add to dashboard Cite

show abstract

“…In [26] Fukui et al presented a novel joint representation technique called MCB for VQA where image and text representation is randomly projected to a higher dimensional space and then join both vectors efficiently by using the element-wise product in Fast Fourier Transform (FFT) space. In [27] authors have used a transformer as a language model and proposed a VQA model with Multiple Parallel Co-attention. They proved that transformers as a language model improve question feature extraction.…”

Section: Related Workmentioning

confidence: 99%

BERT based Hierarchical Alternating Co-Attention Visual Question Answering using Bottom-Up Features

2022

International Journal of Intelligent Systems and Applications in Engineering

View full text Add to dashboard Cite

Answering a question from a given visual image is a very well-known vision language task where the machine is given a pair of an image and a related question and the task is to generate the natural language answer. Humans can easily relate image content with a given question and reason about how to generate an answer. But automation of this task is challenging as it involves many computer vision and NLP tasks. Most of the literature focus on a novel attention mechanism for joining image and question features ignoring the importance of improving the question feature extraction module. Transformers have changed the way spatial and temporal data is processed. This paper exploits the power of Bidirectional Encoder Representation from Transformer (BERT) as a powerful question feature extractor for the VQA model. A novel method of extracting question features by combining output features from four consecutive encoders of BERT has been proposed. This is from the fact that each encoder layer of the transformer attends to features from the word to a phrase and ultimately to a sentence-level representation. A novel BERT-based hierarchical alternating co-attention VQA using the Bottom-up features model has been proposed. Our model is evaluated on the publicly available benchmark dataset VQA v2.0 and experimental results prove that the model improves upon two baseline models by 9.37% and 0.74% respectively.

show abstract

BERT based Multiple Parallel Co-attention Model for Visual Question Answering

Cited by 2 publications

References 7 publications

A Comprehensive Review and Open Challenges on Visual Question Answering Models

A Comprehensive Review and Open Challenges on Visual Question Answering Models

BERT based Hierarchical Alternating Co-Attention Visual Question Answering using Bottom-Up Features

Contact Info

Product

Resources

About