Attention-Mechanism-Containing Neural Networks for High-Resolution Remote Sensing Image Classification

Abstract

A deep neural network is suitable for remote sensing image pixel-wise classification because it effectively extracts features from the raw data. However, remote sensing images with higher spatial resolution exhibit smaller inter-class differences and greater intra-class differences; thus, feature extraction becomes more difficult. The attention mechanism, as a method that simulates the manner in which humans comprehend and perceive images, is useful for the quick and accurate acquisition of key features. In this study, we propose a novel neural network that incorporates two kinds of attention mechanisms in its mask and trunk branches; i.e., control gate (soft) and feedback attention mechanisms, respectively, based on the branches’ primary roles. Thus, a deep neural network can be equipped with an attention mechanism to perform pixel-wise classification for very high-resolution remote sensing (VHRRS) images. The control gate attention mechanism in the mask branch is utilized to build pixel-wise masks for feature maps, to assign different priorities to different locations on different channels for feature extraction recalibration, to apply stress to the effective features, and to weaken the influence of other profitless features. The feedback attention mechanism in the trunk branch allows for the retrieval of high-level semantic features. Hence, additional aids are provided for lower layers to re-weight the focus and to re-update higher-level feature extraction in a target-oriented manner. These two attention mechanisms are fused to form a neural network module. By stacking various modules with different-scale mask branches, the network utilizes different attention-aware features under different local spatial structures. The proposed method is tested on the VHRRS images from the BJ-02, GF-02, Geoeye, and Quickbird satellites, and the influence of the network structure and the rationality of the network design are discussed. Compared with other state-of-the-art methods, our proposed method achieves competitive accuracy, thereby proving its effectiveness.

Attention-Guided Multi-Scale Segmentation Neural Network for Interactive Extraction of Region Objects from High-Resolution Satellite Imagery

“…Since SE-Net [41] only adopts the global average pooling [43] (pp. [29][30][31][32][33][34][35][36][37][38][39], it can encode the entire spatial feature on a channel as a global feature, which is effectively used for standard image segmentation. However, the most important channels in an interactive segmentation task are mostly decided by the user-interactions.…”

“…First, we applied a global average pooling [43] (pp. [29][30][31][32][33][34][35][36][37][38][39] and a global max pooling [43] (pp. [29][30][31][32][33][34][35][36][37][38][39], respectively, to squeeze global spatial information, and obtained the output squeeze gap ∈ R C,1,1 and squeeze gmp ∈ R C,1,1 .…”

“…In our work, we introduced user-interactions into the CNNs to guide the framework to segment the specific target object. Xu et al [32] proposed an ingenious network that incorporates control gate and feedback attention mechanisms to perform pixel-wise classification for satellite imagery. This approach does use ingenious designs to achieve good performance for the automatic satellite imagery interpretation while it needs long processing time.…”

“…With the good performance of interactive image segmentation in the domain of computer vision [33][34][35] and the limitations of the current automatic interpretation [30][31][32], we propose a novel method for deep interactive extraction of non-artificial region objects. In this approach, user-provided interactions (clicks, scribbles) are first transformed into the guidance maps G. Then, we apply a VGG [9] network pre-trained on the ImageNet dataset to the input image X, to effectively extract the feature maps F from the complex background.…”

Moving Learning Machine towards Fast Real-Time Applications: A High-Speed FPGA-Based Implementation of the OS-ELM Training Algorithm

“…Futhermore, even online learning applications with non-stationary and/or imbalanced streaming data can be managed by OS-ELM-based algorithms [33][34][35]. Although, obviously, there are applications that cannot be implemented on FPGA since their limiting resources, as the implementation of attention mechanisms in remote sensing image pixel-wise classification [36], acoustic adaptation models addressing the presence of microphone mismatch in Automatic Speech Recognition (ASR) systems [37], or the prediction of infectious diseases [38].…”