Multi-modal fusion transformer for visual question answering in remote sensing

Abstract

With the new generation of satellite technologies, the archives of remote sensing (RS) images are growing very fast. To make the intrinsic information of each RS image easily accessible, visual question answering (VQA) has been introduced in RS. VQA allows a user to formulate a free-form question concerning the content of RS images to extract generic information. It has been shown that the fusion of the input modalities (i.e., image and text) is crucial for the performance of VQA systems. Most of the current fusion approaches use modalityspecific representations in their fusion modules instead of joint representation learning. However, to discover the underlying relation between both the image and question modality, the model is required to learn the joint representation instead of simply combining (e.g., concatenating, adding, or multiplying) the modality-specific representations. We propose a multi-modal transformer-based architecture to overcome this issue. Our proposed architecture consists of three main modules: i) the feature extraction module for extracting the modality-specific features; ii) the fusion module, which leverages a user-defined number of multi-modal transformer layers of the VisualBERT model (VB); and iii) the classification module to obtain the answer. In contrast to recently proposed transformer-based models in RS VQA, the presented architecture (called VBFusion) is not limited to specific questions, e.g., questions concerning pre-defined objects. Experimental results obtained on the RSVQAxBEN and RSVQA-LR datasets (which are made up of RGB bands of Sentinel-2 images) demonstrate the effectiveness of VBFusion for VQA tasks in RS. To analyze the importance of using other spectral bands for the description of the complex content of RS images in the framework of VQA, we extend the RSVQAxBEN dataset to include all the spectral bands of Sentinel-2 images with 10m and 20m spatial resolution. Experimental results show the importance of utilizing these bands to characterize the land-use land-cover classes present in the images in the framework of VQA. The code of the proposed method is publicly available at https://git.tu-berlin.de/rsim/multimodal-fusion-transformer-for-vqa-in-rs.

Transformers for Remote Sensing: A Systematic Review and Analysis

“…In CV, transformers have demonstrated advantages in processing multimodal data due to their more general and flexible modeling space [ 74 ]. Consequently, researchers started employing transformers to address multimodal problems in RS image text retrieval [ 75 ] and RS visual question answering [ 76 ]. Currently, there exists a scarcity of large-scale multimodal datasets, leading to researchers’ need to collect multimodal data by themselves.…”

A Spatial Hierarchical Reasoning Network for Remote Sensing Visual Question Answering

“…These prompts were then input into a language model for answer prediction. Siebert et al [21] employed the VisualBERT [7] model to better learn joint representation.…”

“…It is worth noting that Bazi et al [19] introduced an encoderdecoder structure, increasing the model's complexity, while Siebert et al [21] performed full fine-tuning on VisualBERT [7], requiring substantial computational resources and runtime. Compared to these existing transformer-based methods, RSAdapter achieves efficient fine-tuning without increasing model complexity, saving training time and computational resources.…”

“…These features were then separately fed into two parallel decoder transformers to learn a joint representation. Siebert et al [21] fine-tuned datasets using the VisualBERT [7] model.…”

Visual Question Generation From Remote Sensing Images

“…The results indicate that more complex fusion strategies yield higher accuracies. More recently, Siebert et al [37], proposed a VQA model that uses a multi-modal fusing module based on VisualBERT to integrate the image and the language modalities.…”