The semantic segmentation and understanding of remote sensing images applying computer technology has become an important component of monitoring mangrove forests’ ecological changes due to the rapid advancement of remote sensing technology. To improve the semantic segmentation capability of various surface features, this paper proposes a semantic understanding method for mangrove remote sensing images based on convolution feature inference. Firstly, the sample data is randomly selected, and next a model of convolution feature extraction is used to obtain the features of the selected sample data and build an initial feature set. Then, the convolution feature space and rule base are generated by establishing the three-dimensional color space distribution map for each class and domain similarity is introduced to construct the feature set and rules for reasoning. Next, a confidence reasoning method based on the convolution feature region growth, which introduces an improved similarity calculation, is put forward to obtain the first-time reasoning results. Finally, this approach adds a correction module, which removes the boundary information and reduces the noise from the results of the first-time reasoning as a new sample to correct the original feature set and rules, and uses the corrected feature set and rules for reasoning and understanding to obtain the final image segmentation results. It uses the corrected feature set and rules for reasoning and understanding to obtain the final image segmentation results. Experiments show that this algorithm has the benefits of a simple process, a short training time, and easy feature acquisition. The effect has been obviously improved compared to a single threshold segmentation method, K-Nearest Neighbor (KNN), Support Vector Machine (SVM), and other image segmentation methods.