Aerial scene understanding in the wild: Multi-scene recognition via prototype-based memory networks

Abstract

Aerial scene recognition is a fundamental visual task and has attracted an increasing research interest in the last few years. Most of current researches mainly deploy efforts to categorize an aerial image into one scene-level label, while in real-world scenarios, there often exist multiple scenes in a single image. Therefore, in this paper, we propose to take a step forward to a more practical and challenging task, namely multi-scene recognition in single images. Moreover, we note that manually yielding annotations for such a task is extraordinarily time- and labor-consuming. To address this, we propose a prototype-based memory network to recognize multiple scenes in a single image by leveraging massive well-annotated single-scene images. The proposed network consists of three key components: 1) a prototype learning module, 2) a prototype-inhabiting external memory, and 3) a multi-head attention-based memory retrieval module. To be more specific, we first learn the prototype representation of each aerial scene from single-scene aerial image datasets and store it in an external memory. Afterwards, a multi-head attention-based memory retrieval module is devised to retrieve scene prototypes relevant to query multi-scene images for final predictions. Notably, only a limited number of annotated multi-scene images are needed in the training phase. To facilitate the progress of aerial scene recognition, we produce a new multi-scene aerial image (MAI) dataset. Experimental results on variant dataset configurations demonstrate the effectiveness of our network. Our dataset and codes are publicly available 1 .

PCINet: a Prototype- and Concept-based Interpretable Network for Mutli-scene Recognition

“…The feature map X is then compared with a set of predefined scene prototypes P = [p1, p2, ..., pn] T , where N is the number of scene categories and pi denotes the prototype of the i-th scene. In this work, we follow (Hua et al, 2021a) and generate scene prototypes by first training f ϕ on a single-scene aerial image dataset and then summarizing features of samples belonging to the i-th scene as its prototype pi. Thus, pi is expected to be representative of its correspond- ing scene (see Figure 5).…”

Classification of Scenes in Aerial Images with Deep Learning Models

“…For this reason, researchers in the literature have initially created benchmark data sets for artificial intelligence-based vision systems to be developed in this field [1][2][3][4][5]. Deep learning-based methods were developed on these datasets [6][7][8][9][10][11][12]. Deep learning is a sub-branch of artificial intelligence and first attracted attention with the ImageNet competition in 2012.…”

“…The SCDAL framework is tested on two large aerial image datasets and shown to be superior to most existing domain adaptation methods with at least a 3% improvement in overall accuracy. Work [8] proposes a prototype-based memory network for recognizing multiple scenes in a single aerial image. The network consists of a prototype learning module, a prototype-hosting external memory, and a multi-head attention-based memory retrieval module.…”

Self-Supervised Learning in Remote Sensing: A review

“…Within the remote sensing community, semi-supervised learning has been long studied and enjoys applications in, e.g., hyperspectral image recognition and processing [19,20,21,22,23,24,25,26,27,28], multi-spectral image segmentation [29,30,31,32,33,34,35] and SAR-optical data fusion [36].…”