Remote Sensing Image Captioning via Multilevel Attention-Based Visual Question Answering

Murali, Nirmala; Shanthi, A. P.

doi:10.1007/978-981-19-0475-2_41

Cited by 5 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Wang et al [35] combined features from a multi-labeling model with CNN-extracted features to train a captioning model. Murali et al [36] proposed a captioning model that initially undergoes training on VQA as an auxiliary task, followed by leveraging the acquired knowledge to generate more accurate captions. In [37], a multi-label classifier is employed to generate labels from the image, which are then utilized, along with ground truth captions, to train the captioning model.…”

Section: Multi-tasking In Nlp For Rsmentioning

confidence: 99%

RS-LLaVA: A Large Vision-Language Model for Joint Captioning and Question Answering in Remote Sensing Imagery

Bazi,

Bashmal,

Al Rahhal

et al. 2024

Remote Sensing

View full text Add to dashboard Cite

In this paper, we delve into the innovative application of large language models (LLMs) and their extension, large vision-language models (LVLMs), in the field of remote sensing (RS) image analysis. We particularly emphasize their multi-tasking potential with a focus on image captioning and visual question answering (VQA). In particular, we introduce an improved version of the Large Language and Vision Assistant Model (LLaVA), specifically adapted for RS imagery through a low-rank adaptation approach. To evaluate the model performance, we create the RS-instructions dataset, a comprehensive benchmark dataset that integrates four diverse single-task datasets related to captioning and VQA. The experimental results confirm the model’s effectiveness, marking a step forward toward the development of efficient multi-task models for RS image analysis.

show abstract

Section: Multi-tasking In Nlp For Rsmentioning

confidence: 99%

RS-LLaVA: A Large Vision-Language Model for Joint Captioning and Question Answering in Remote Sensing Imagery

Bazi,

Bashmal,

Al Rahhal

et al. 2024

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…( 1) and weight normalization coefficient is calculated in Eq. (2). The edge recovery is an iterative process which is mathematically expressed in Eq.…”

Section: Data Preprocessingmentioning

confidence: 99%

“…Traditional remote sensing jobs frequently focus on low-level semantic data via image synthesis. [2]. A word level label is given to an RSI by image classification to exchange the low-level semantic information.…”

Section: Introductionmentioning

confidence: 99%

Deep Attention Based Dense Net with Visual Switch Added BiLSTM for Caption Generation from Remote Sensing Images

2023

IJIES

View full text Add to dashboard Cite

Remote sensing image captioning is the challenging task due to low global information, single feature extraction and lack of detailed image captions. To address these issues, this research proposed a deep attention based DenseNet with visual switch added bidirectional long short-term memory (DADN-BiLSTM) for captioning. In this research, initially the images and captions are collected from captioning dataset to smooth away small structures. After that, a double attention mechanism is applied to DenseNet for capturing weak features and to improve the problem corresponds between image feature and captioning information. At the same time, a clustering-based segmentation is more useful and easier to segment the image as smaller parts to make the access easily. Moreover, a decoder is used to improve the use of captioning context information. Then the proposed system is implemented in PYTHON and the performance is evaluated against existing methods in terms of some relevant evaluation metrics such as, recall-oriented understudy for gisting evaluation, accuracy and bilingual evaluation understudy. Finally, the experimental results achieve higher scores in all evaluation indicators such as 0.8925 BLEU1, 0.8514 BLEU2, 0.8252 BLEU3, 0.8312 BLEU4 and 0.8611 ROUGE score on UCM captions, 0.8532 BLEU1, 0.7912 BLEU2, 0.8351 BLEU3, 0.7215 BLEU4 and 0.8139 ROUGE score on Sydney captions and 0.8125 BLEU1, 0.7501 BLEU2, 0.6812 BLEU3, 0.7254 BLEU4 and 0.8245 ROUGE score on RSICD captions.

show abstract

“…The two-phase VQA model was proposed in Ref. 39, where three levels of attention are used in the first phase to identify the important words, which is then used in second phase to enhance the captions. The generated captions are informative enough as they include all the image objects and their count.…”

Section: Visual Question Answeringmentioning

confidence: 99%

Graph neural network-based visual relationship and multilevel attention for image captioning

Sharma

Srivastava

2022

J. Electron. Imag.

View full text Add to dashboard Cite

With the remarkable success of the image captioning tasks, visual attention methods have become a vital part of captioning models. However, most attention-based image captioning methods do not consider any relationship among regions, which play a significant role in better image understanding. We proposed an image captioning method based on local relation network using a multilevel attention approach with graph neural network. It not only fully explores the relationship between the object and the image regions but also generates significant and contextbased features corresponding to every region in the image. The attention employed in our work enhances the image representation capability of our method by focusing on a given image region and its related image regions. Thus addressing the relevant contextual information, spatial locations, and deep visual features leads to improve caption generation. We verified the effectiveness of the proposed model by conducting extensive experiments on three benchmark datasets: Flickr30k, MSCOCO, and nocaps. The results show the superiority of the proposed method over the existing methods both in quantitative and qualitative manners. Detailed ablation studies are conducted to communicate how each part would contribute to the final performance.

show abstract

Remote Sensing Image Captioning via Multilevel Attention-Based Visual Question Answering

Cited by 5 publications

References 21 publications

RS-LLaVA: A Large Vision-Language Model for Joint Captioning and Question Answering in Remote Sensing Imagery

RS-LLaVA: A Large Vision-Language Model for Joint Captioning and Question Answering in Remote Sensing Imagery

Deep Attention Based Dense Net with Visual Switch Added BiLSTM for Caption Generation from Remote Sensing Images

Graph neural network-based visual relationship and multilevel attention for image captioning

Contact Info

Product

Resources

About