Semantic Segmentation of Aerial Imagery via Multi-Scale Shuffling Convolutional Neural Networks With Deep Supervision

Abstract

<p><strong>Abstract.</strong> In this paper, we address the semantic segmentation of aerial imagery based on the use of multi-modal data given in the form of true orthophotos and the corresponding Digital Surface Models (DSMs). We present the Deeply-supervised Shuffling Convolutional Neural Network (DSCNN) representing a multi-scale extension of the Shuffling Convolutional Neural Network (SCNN) with deep supervision. Thereby, we take the advantage of the SCNN involving the shuffling operator to effectively upsample feature maps and then fuse multiscale features derived from the intermediate layers of the SCNN, which results in the Multi-scale Shuffling Convolutional Neural Network (MSCNN). Based on the MSCNN, we derive the DSCNN by introducing additional losses into the intermediate layers of the MSCNN. In addition, we investigate the impact of using different sets of hand-crafted radiometric and geometric features derived from the true orthophotos and the DSMs on the semantic segmentation task. For performance evaluation, we use a commonly used benchmark dataset. The achieved results reveal that both multi-scale fusion and deep supervision contribute to an improvement in performance. Furthermore, the use of a diversity of hand-crafted radiometric and geometric features as input for the DSCNN does not provide the best numerical results, but smoother and improved detections for several objects.</p>

Npaloss: Neighboring Pixel Affinity Loss for Semantic Segmentation in High-Resolution Aerial Imagery

“…Inference Strategy: Due to the large difference between the size of the tiles and the training size of the network, using the entire image directly as an input to the network may reduce prediction accuracy. Therefore, we choose the sliding window strategy (Chen et al, 2018b;Fu et al, 2020) when making inference. Specifically, we crop the patches from the tiles in an overlapping manner and set the overlapping stride to 1/3.…”

Deep Learning-Based Semantic Segmentation of Urban Features in Satellite Images: A Review and Meta-Analysis

“…Different approaches of data fusion such as multi-modal, multi-scale and multikernel data fusion have also been practiced to improve the performance of FCN on RS images [56,80,[91][92][93][94][95][96][97][98]. In a distinctive approach, some studies focused on the symmetry of the encoder-decoder structure, which is discussed in the next subsection.…”

“…The most commonly performed methods calculation of DSM, nDSM, LiDAR data and NDVI [76,84,92,94,146,150]. Some used a stack of multiple channels of images [95,101,144,160]. Some used filters like unsharp mask filter, median filter, linear contrast filter [106] and Wiener filter [139].…”

“…The distribution of these methods is shown in Figure 3. Some also evaluated the increment in performance due to augmentation [95]. More details on this section can be found in Appendix A.…”

“…Liu et al (2020)[107] used two discriminators to minimize the discrepancy between the source and target domains. Also, many others performed multi-modal, multi-scale and multi-resolution training of DL-models[56,80,[91][92][93][94][95][96][97][98], for sufficient training of DL model.…”