Position-Sensitive Attention Based on Fully Convolutional Neural Networks for Land Cover Classification

Abstract

Abstract. Pixel-wise land cover classification is a fundamental task in remote sensing image interpretation, aiming to identify planimetric features (e.g., trees, waters, buildings etc.) from earth's surface. Recently, deep learning methods based on fully convolutional neural networks (FCN) become the mainstream approach for land cover classification, thanks to their superior performance in the image context perception and features learning. However, for high-resolution remote sensing images with huge quantity of object details, some deep learning based methods often ignore many important details by nature, specially, in the procedure of pooling operation and stacking convolutions in conventional FCN, it can leads to ambiguous classification of adjacent objects. To refine lost details caused by the stacking convolutions, we propose a position-sensitive attention (PSA) based on skip connections for land cover classification with high-resolution remote sensing images, which designs to deliver a weight that is sensitive to the spatial details in remote sensing images, the PSA module is able to improve pixel-level details scattered across spatial positions. Experimental results demonstrate that our method can be feasible to existing FCN-based models, 1% improvement in F1-score is obtained on 2021 "Shengteng Cup" competition dataset after using PSA, when comparing to several state-of-the-art methods, similar or even better performance is achieved on the ISPRS Vaihingen 2D dataset, but with less parameters.

Multiscale Feature Reconstruction and Interclass Attention Weighting for Land Cover Classification

“…In general, the strategies of reintroducing details also deliver ambiguous features [42] , post-processing methods rely on coarse segmentation results, and sub-networks of edge detection increase the computation cost and require ground truth of boundary for loss evaluation. Based on our previous work on PSA model [3] , we incorporate PSA model into highresolution details refinement as a part of our Multi-Scale Feature Reconstruction (MSFR) architecture, which deal with the multi-scale feature and lost details at the same time.…”

“…The MSFR architecture takes an image as input, and output a quarter-sized feature map which will act as the input for the ICAW architecture. MSFR can be deployed in different serial encoder and decoder structures, in which we generate the Position-Sensitive Attention (PSA) with the feature maps from encoder at the highest resolution (number 1 in blue circle) [3] and the Multi-Scale Channel Attention (MCA) at the rest two resolutions (numbers 2 and 3 in blue circles). Based on PSA and MCA, we weight the corresponding feature maps from decoder to provide multi-scale information with expectation for recovering more details.…”

“…s a significant component of remote sensing image interpretation, land cover classification aims to parse each pixel into meaningful category labels [1][2] [3] . In the early time, land cover classification was explored by indexbased methods, such as normalized vegetation index NDVI (Normalized Difference Vegetation Index) [4] ,built-up area presence index PanTex [5] etc.…”

“…In CNN-based methods, the feature maps from the highresolution layers typically have good focus on the tiny objects, but observe large objects partly because of the smaller receptive fields, while low-resolution feature maps with larger receptive fields ignore details and have difficulties to identify adjacent objects in tiny sizes [19] . Another limitation by CNN methods is the lost details caused by down-sampling operations in the encoding stages [2] [3] , which negatively affects the classification of small objects as well as edges [32] (shown by Fig. 1(a)).…”

“…To cope with the mentioned limitations, based on serial learning networks (ResNet [38] for example) whose features are consecutively encoded from the high-resolution feature maps to 2 > REPLACE THIS LINE WITH YOUR MANUSCRIPT ID NUMBER (DOUBLE-CLICK HERE TO EDIT) < the low-resolution feature maps through a single path, we proposed two corresponding pluggable modules: first, to take care of both multi-scale variance and lost details, we propose a Multi-Scale Feature Reconstruction (MSFR) module consisting of two models. More specifically, our previous Position Sensitive Attention (PSA) [3] model is run on the highest resolution feature maps which are more sensitive to local edges, then, a new Multi-Scale Channel Attention (MCA) is further applied to reconstruct multi-scale feature representation for the lower-resolution features. Second, to cope with erroneous classification caused by intra-class differences and inter-class correlations, we present an Inter-Class Attention Weighting (ICAW) module, it first generates feature vectors for each category and applies the multi-head attention mechanism from Transformer [39] [40] to investigate a weighted fusion of inter-class distances based on per-class feature vectors.…”