Mangroves are special vegetation that grows in the intertidal zone of the coast and has extremely high ecological and environmental value. Different mangrove species exhibit significant differences in ecological functions and environmental responses, so accurately identifying and distinguishing these species is crucial for ecological protection and monitoring. However, mangrove species recognition faces challenges, such as morphological similarity, environmental complexity, target size variability, and data scarcity. Traditional mangrove monitoring methods mainly rely on expensive and operationally complex multispectral or hyperspectral remote sensing sensors, which have high data processing and storage costs, hindering large-scale application and popularization. Although hyperspectral monitoring is still necessary in certain situations, the low identification accuracy in routine monitoring severely hinders ecological analysis. To address these issues, this paper proposes the UrmsNet segmentation network, aimed at improving identification accuracy in routine monitoring while reducing costs and complexity. It includes an improved lightweight convolution SCConv, an Adaptive Selective Attention Module (ASAM), and a Cross-Layer Feature Fusion Module (CLFFM). ASAM adaptively extracts and fuses features of different mangrove species, enhancing the network’s ability to characterize mangrove species with similar morphology and in complex environments. CLFFM combines shallow details and deep semantic information to ensure accurate segmentation of mangrove boundaries and small targets.Additionally, this paper constructs a high-quality RGB image dataset for mangrove species segmentation to address the data scarcity problem. Compared to traditional methods, our approach is more precise and efficient. While maintaining relatively low parameters and computational complexity (FLOPs), it achieves excellent performance with mIoU and mPA metrics of 92.21% and 95.98%, respectively. This performance is comparable to the latest methods using multispectral or hyperspectral data but significantly reduces cost and complexity. By combining periodic hyperspectral monitoring with UrmsNet-supported routine monitoring, a more comprehensive and efficient mangrove ecological monitoring can be achieved.These research findings provide a new technical approach for large-scale, low-cost monitoring of important ecosystems such as mangroves, with significant theoretical and practical value. Furthermore, UrmsNet also demonstrates excellent performance on LoveDA, Potsdam, and Vaihingen datasets, showing potential for wider application.