Visual Question Generation From Remote Sensing Images

Abstract

Visual question generation (VQG) is a fundamental task in vision-language understanding that aims to generate relevant questions about the given input image. In this paper, we propose a paragraph-based VQG approach for generating intelligent questions in natural language about remote sensing (RS) images. Specifically, our proposed framework consists of two transformer-based vision and language models. First, we employ a swin-transformer encoder to generate a multi-scale representative visual feature from the image. Then, this feature is used as a prefix to guide a generative pre-trained transformer-2 (GPT-2) decoder in generating multiple questions in the form of a paragraph to cover the abundant visual information contained in the RS scene. To train the model, the language decoder is finetuned on RS dataset to generate a set of relevant questions from the RS image. We evaluate our model on two visual questionanswering (VQA) datasets in RS. Additionally, we construct a new dataset termed TextRS-VQA for better evaluation for our VQG model. This dataset consists of questions completely annotated by humans which addresses the high redundancy of the questions in prior VQA datasets. Extensive experiments using several accuracy and diversity metrics demonstrate the effectiveness of our proposed VQG model in generating meaningful, valid, and diverse questions from RS images. The TextRS-VQA dataset is available at: https://github.com/yakoubbazi/TextRS.

Machine-to-Machine Visual Dialoguing with ChatGPT for Enriched Textual Image Description

“…From what we know, ref. [26] is the only work in the literature exploring generative-based visual question generation for RS images. In [26] the authors propose a transformer-based architecture to generate different plausible questions for RS images.…”

“…[26] is the only work in the literature exploring generative-based visual question generation for RS images. In [26] the authors propose a transformer-based architecture to generate different plausible questions for RS images. They use questions from the dataset proposed in [15] to train the model in a fully supervised fashion.…”

Potential Impact of Using ChatGPT-3.5 in the Theoretical and Practical Multi-Level Approach to Open-Source Remote Sensing Archaeology, Preliminary Considerations

“…Since ChatGPT was released, the scientific community has been using it, and articles have been published on it [107] on several topics [106,[110][111][112][113][114][115][116][117][118][119][120], as well as ethical issues that have arisen very recently [121][122][123][124][125][126]. To date, there are few studies that demonstrate the usefulness of GPT or derived tools (e.g., Visual ChatGPT) in the field of RS and satellite image classification [127][128][129][130]. A useful tool made available by the world community via the web (e.g., GitHub or several Google extensions) is the possibility of being able to use prompts (i.e., texts explaining to ChatGPT what to do) that are already pre-compiled so as to (i) save time and (ii) prevent the system from being trained wrongly or giving wrong answers.…”

Brain-Inspired Remote Sensing Foundation Models and Open Problems: A Comprehensive Survey

“…Yuan et al [232] introduce a self-paced curriculum learning approach for VQA in remote sensing. Models like BERT [231,235], CLIP [233], and GPT [237] have been widely applied, and addressing the open-set problem of VQA in remote sensing is also a focus of research [234]. The VQA task related to change detection is an emerging research direction [236].…”