From Easy to Hard: Learning Language-Guided Curriculum for Visual Question Answering on Remote Sensing Data

Yuan, Zhenghang; Mou, Lichao; Wang, Qi; Zhu, Xiao Xiang

doi:10.1109/tgrs.2022.3173811

Cited by 28 publications

(21 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, AOA [33] handles the common-seen question better than the others (e.g., Number and Yes/No are classical questions for natural images), while our method performs better in the positioning related questions (e.g., Size, Location, Shape). (3) CNN models [3,25,43,47] incline to perform better than transformers [33,45,50] in the term of Color questions, which is nearly 25% gap. On the other hand, these questions are always about the information of isolated objects (e.g., small vehicle, large vehicle, ship), which are significantly smaller than the others.…”

Section: Benchmark Resultsmentioning

confidence: 99%

“…We could see the significant improvements with our proposed GFTransformer. Specifically, compared to the state-of-the-art VQA model designed for aerial images [47], 2.45% overall accuracy and 2.70% average accuracy boosts are obtained on Test-Phila. Besides, we perform better than the models designed for natural scenes [3,29,33,43,50] in general with 1.53% overall accuracy and 1.74% average accuracy improvements.…”

Section: Results On Rsvqamentioning

confidence: 99%

“…GQA [16] presents natural images with scene graphs and questions requiring diverse reasoning skills and sequential steps to generate answers. With respect to the field of earth observation, there have also been some pioneers delving into VQA [34,47]. RSVQA [25] consists of one low-resolution subset and one high-resolution subset with simple QA pairs.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

HRVQA: A Visual Question Answering Benchmark for High-Resolution Aerial Images

Li¹,

Vosselman²,

Yang³

2023

Preprint

View full text Add to dashboard Cite

Visual question answering (VQA) is an important and challenging multimodal task in computer vision. Recently, a few efforts have been made to bring VQA task to aerial images, due to its potential real-world applications in disaster monitoring, urban planning, and digital earth product generation. However, not only the huge variation in the appearance, scale and orientation of the concepts in aerial images, but also the scarcity of the well-annotated datasets restricts the development of VQA in this domain. In this paper, we introduce a new dataset, HRVQA, which provides collected 53512 aerial images of 1024 × 1024 pixels and semiautomatically generated 1070240 QA pairs. To benchmark the understanding capability of VQA models for aerial images, we evaluate the relevant methods on HRVQA. Moreover, we propose a novel model, GFTransformer, with gated attention modules and a mutual fusion module. The experiments show that the proposed dataset is quite challenging, especially the specific attribute related questions. Our method achieves superior performance in comparison to the previous state-of-the-art approaches. The dataset and the source code will be released at https://hrvqa.nl/.

show abstract

Section: Benchmark Resultsmentioning

confidence: 99%

Section: Results On Rsvqamentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

HRVQA: A Visual Question Answering Benchmark for High-Resolution Aerial Images

Li¹,

Vosselman²,

Yang³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…If we can provide an answer to the above question in natural language instead of maps, it will make users understand better. In this context, tasks of combining remote sensing images and natural language, such as image captioning [8,9] and visual question answering (VQA) [10,11], attract a lot of attention in the remote sensing community nowadays.…”

Section: Introductionmentioning

confidence: 99%

Change-Aware Visual Question Answering

Yuan

Mou

Zhu

2022

IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium

Self Cite

View full text Add to dashboard Cite

Change detection has been a hot research topic in the field of remote sensing, and it can provide information on observing changes of Earth's surface. However, segmentation-based change results are not very friendly to end users. Thus, in order to improve user experience and offer them high-level semantic information on change detection, we introduce a new task: change-aware visual question answering (VQA) on multi-temporal aerial images. Specifically, given a pair of multi-temporal aerial images and questions, this task aims to automatically provide natural language answers. By doing so, end users have better access to easy-to-understand change information through natural language. Besides, we also create a dataset made of multi-temporal image-question-answer triplets and a baseline method for this task. Experimental results offer valuable insights for the further research on this task.

show abstract

“…Recently, a multilayer aggregated Transformer was utilized to extract information for caption generation [30]. Regarding VQA for remote sensing data (RSVQA), Lobry et al [31] first introduced this task, built two datasets, and used a hybrid CNN-RNN model to extract feature, and Yuan et al [32] proposed a self-paced curriculum learning based model trained from easy to hard questions gradually.…”

mentioning

confidence: 99%