With the rapid socioeconomic development in China, the increasing soil erosion caused by anthropogenic production and construction activities is taking place, which is characterized by short duration, high frequency, and great damage to its surrounding environment. Therefore, the regulation and control of soil erosion of anthropogenically disturbed parcels is an urgent task. This study proposes an improved model that combines the boundary constraint and jagged hybrid dilated convolution channel shuffling module (BCJHDC) and the polarized self-attention (PSA) module for extracting anthropogenically disturbed parcels with soil erosion (ADPSE) from high-resolution remote sensing images in Hubei Province. Firstly, the PSA module is added to the encoder to better extract the feature information of the target object. Secondly, BCJHDC module is used to extract multi-scale semantic information from images and improve the boundary segmentation quality. Precision, recall, intersection over union (IOU), and F1 score are calculated to evaluate the model accuracy. The results indicate that our improved model performs well on the human-perturbed parcel extraction task with an F1 of 87.92% and an IOU of 78.44%. Ablation experiments and application experiments suggest the validity the applicability, and the portability of our proposed improved model, respectively. Compared with the other 7 advanced semantic segmentation models, our improved model has significant advantages. Overall, this study provides valuable reference for policy formulation of water and soil conservation.