ClassHyPer: ClassMix-Based Hybrid Perturbations for Deep Semi-Supervised Semantic Segmentation of Remote Sensing Imagery

Abstract

Inspired by the tremendous success of deep learning (DL) and the increased availability of remote sensing data, DL-based image semantic segmentation has attracted growing interest in the remote sensing community. The ideal scenario of DL application requires a vast number of annotation data with the same feature distribution as the area of interest. However, obtaining such enormous training sets that suit the data distribution of the target area is highly time-consuming and costly. Consistency-regularization-based semi-supervised learning (SSL) methods have gained growing popularity thanks to their ease of implementation and remarkable performance. However, there have been limited applications of SSL in remote sensing. This study comprehensively analyzed several advanced SSL methods based on consistency regularization from the perspective of data- and model-level perturbation. Then, an end-to-end SSL approach based on a hybrid perturbation paradigm was introduced to improve the DL model’s performance with a limited number of labels. The proposed method integrates the semantic boundary information to generate more meaningful mixing images when performing data-level perturbation. Additionally, by using implicit pseudo-supervision based on model-level perturbation, it eliminates the need to set extra threshold parameters in training. Furthermore, it can be flexibly paired with the DL model in an end-to-end manner, as opposed to the separated training stages used in the traditional pseudo-labeling. Experimental results for five remote sensing benchmark datasets in the application of segmentation of roads, buildings, and land cover demonstrated the effectiveness and robustness of the proposed approach. It is particularly encouraging that the ratio of accuracy obtained using the proposed method with 5% labels to that using the purely supervised method with 100% labels was more than 89% on all benchmark datasets.

Enhancement of Urban Floodwater Mapping From Aerial Imagery With Dense Shadows via Semisupervised Learning

“…It is noted that the parameters of VGG-16 and ResNet-50 were obtained from the official pretrained model on the ImageNet dataset [45]. While for the SSL pipeline, two prominent SSL frameworks, including CPS [53], and ClassHyPer [33] were adopted for comparison. Of which, CPS performs slight parameter perturbations between two identical base models and imposes pseudosupervision on each other for unsupervised learning; on the strength of CPS, ClassHyPer integrates ClassMix to improve the model capability.…”

“…In particular, a few studies [31], [32] demonstrated the superiority of DL algorithms for shadow detection from RGB aerial imagery, making us confident that DL can cope with shadow problems in urban floodwater mapping. However, a huge demand for labeled training data still poses challenges in the DL domain [33]. Moreover, most relevant studies aimed to train a DL model with large quantities of labeled samples in a single location, but they may fail to generalize the same model across unseen locations due to the varying illumination and atmospheric conditions [15], [34].…”

A Survey of Deep Learning Road Extraction Algorithms Using High-Resolution Remote Sensing Images

“…This approach not only improved the classification accuracy of semantic segmentation networks but also mitigated the negative impact of limited labeled data on network performance. In another study [ 148 ], which focused on consistency regularization in semi-supervised learning, perturbation schemes were reviewed, and prominent data-level perturbation schemes, CutMix and ClassMix (a development from CutMix), as well as model-level perturbation representatives, mean teacher (MT) and cross pseudo-supervision (CPS), were identified. Inspired by these four perturbation methods, an end-to-end semi-supervised semantic segmentation framework named “ClassHyPer” was proposed.…”

Multimodal Colearning Meets Remote Sensing: Taxonomy, State of the Art, and Future Works

“…With perturbation-based methods, the learning process assumes that model predictions should be robust to noise. The objective function ensures that encoded feature spaces are minimally affected by a small amount of noise, as seen in studies such as [131], [132]. Manifold methods, on the other hand, operate under the assumption that some small variations in the input can yield different results.…”