Adaptive attention fusion network for visual question answering

Gu, Geonmo; Kim, Seong Tae; Ro, Yong Man

doi:10.1109/icme.2017.8019540

Cited by 5 publications

(6 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many approaches to VQA have been explored in the past. For example, [4,5,27,28] used attention-based mechanisms to solve VQA problems. Instead of attending to every detail of the image and every word of the inquiry, an attention mechanism enables us to concentrate solely on the most crucial portions of the image and questions.…”

Section: Related Workmentioning

confidence: 99%

“…Different approaches have been experimented with for achieving each of the steps in the literature. For image feature extraction, recent deep learning Models like VGGNet [1][2][3][4][5][6], ResNet [7][8][9], and even F-RCNN [10][11][12] have proved to be a boon. For question feature extraction, literature explored various word embedding techniques ranging from simple word2vec [13,14], and Glove [15,16] to complex LSTM, GRU [14,15,17], and transformers [18][19][20].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

EDUVQA – Visual Question Answering: An Educational Perspective

Dipali Koshti,

Ashutosh Gupta,

Mukesh Kalla

et al. 2024

ARASET

View full text Add to dashboard Cite

Increasing applications of artificial intelligence in the field of education have changed the way school children learn various concepts. Educational Visual Question Answering or EDUVQA is one such application that allows students to interact directly with images, ask educational questions, and get the correct answer. Two major challenges faced by educational VQA are the lack of availability of domain-specific datasets and often it requires referring to the external knowledge bases to answer open-domain questions. We propose a novel EDUVQA model developed especially for educational purposes and introduce our own EDUVQA dataset. The dataset consists of four categories of images - animals, plants, fruits, and vegetables. The majority of the currently used techniques focus on the extraction of picture and question characteristics in order to discover the joint feature embeddings via multimodal fusion or attention mechanisms. We propose a different method that aims to better utilize the semantic knowledge present in images. Our approach entails building an EDUVQA dataset using educational images, where each data point is made up of an image, a question that corresponds to it, a valid response, and a fact that supports it. The fact is created in the form of <S,V,O> triplet where ‘s’ denotes a subject, ‘v’ a verb, and ‘o’ an object. First, an SVO detector model is trained on EDUVQA dataset capable of predicting the Subject, Verb, and Object present in the image-question pair. Using this <S,V,O> triplet, the most relevant facts from our fact base are extracted. The final answer is predicted using these extracted facts, image, and question attributes. The image features are extricated using pretrained ResNet and question features using a pre-trained BERT model. We have optimized and improved on the current methodologies that use a Relation-based approach and built our SVO-detector model that outperforms current models by 10%.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

EDUVQA – Visual Question Answering: An Educational Perspective

Dipali Koshti,

Ashutosh Gupta,

Mukesh Kalla

et al. 2024

ARASET

View full text Add to dashboard Cite

show abstract

“…Most VQA literature utilizes Convolution Neural Networks (CNN) and their variants for image featurization. In the beginning, many researchers used VGGNet for the extraction of image features [1,2,3,4,5]. They used the final hidden layer of VGG-Net as image features as most of the spatial information is retained in the last hidden layer.…”

Section: Feature Extractionmentioning

confidence: 99%

“…3 We critically analyze all the state-of-the-art (SoTa), end-to-end models, for VQA, their limitations, and future improvements. 4 Finally, we provide the guidelines and future direction for further improvements in VQA models.…”

Section: Introductionmentioning

confidence: 99%

A Comprehensive Review and Open Challenges on Visual Question Answering Models

Shaik,

Koshti,

Gupta

et al. 2023

AiBi Revista de Investigación, Administración e Ingeniería

View full text Add to dashboard Cite

Users are now able to actively interact with images and pose different questions based on images, thanks to recent developments in artificial intelligence. In turn, a response in a natural language answer is expected. The study discusses a variety of datasets that can be used to examine applications for visual question-answering (VQA), as well as their advantages and disadvantages. Four different forms of VQA models—simple joint embedding-based models, attention-based models, knowledge-incorporated models, and domain-specific VQA models—are in-depth examined in this article. We also critically assess the drawbacks and future possibilities of all current state-of-the-art (SoTa), end-to-end VQA models. Finally, we present the directions and guidelines for further development of the VQA models.

show abstract

“…The model was evaluated on VQA 1.0 dataset and they achieved an accuracy of 62.1%. A similar hierarchical approach is used in [25] with novel multi-step reasoning and adaptive fusion method. Here, the model performs textual and, visual attention through two steps.…”

Section: Related Workmentioning

confidence: 99%

BERT based Hierarchical Alternating Co-Attention Visual Question Answering using Bottom-Up Features

2022

International Journal of Intelligent Systems and Applications in Engineering

View full text Add to dashboard Cite

Answering a question from a given visual image is a very well-known vision language task where the machine is given a pair of an image and a related question and the task is to generate the natural language answer. Humans can easily relate image content with a given question and reason about how to generate an answer. But automation of this task is challenging as it involves many computer vision and NLP tasks. Most of the literature focus on a novel attention mechanism for joining image and question features ignoring the importance of improving the question feature extraction module. Transformers have changed the way spatial and temporal data is processed. This paper exploits the power of Bidirectional Encoder Representation from Transformer (BERT) as a powerful question feature extractor for the VQA model. A novel method of extracting question features by combining output features from four consecutive encoders of BERT has been proposed. This is from the fact that each encoder layer of the transformer attends to features from the word to a phrase and ultimately to a sentence-level representation. A novel BERT-based hierarchical alternating co-attention VQA using the Bottom-up features model has been proposed. Our model is evaluated on the publicly available benchmark dataset VQA v2.0 and experimental results prove that the model improves upon two baseline models by 9.37% and 0.74% respectively.

show abstract

Adaptive attention fusion network for visual question answering

Cited by 5 publications

References 11 publications

EDUVQA – Visual Question Answering: An Educational Perspective

EDUVQA – Visual Question Answering: An Educational Perspective

A Comprehensive Review and Open Challenges on Visual Question Answering Models

BERT based Hierarchical Alternating Co-Attention Visual Question Answering using Bottom-Up Features

Contact Info

Product

Resources

About