Image segmentation is an important task in image processing and analysis but due to the same ground object having different spectra and different ground objects having similar spectra, segmentation, particularly on high-resolution remote sensing images, can be significantly challenging. Since the spectral distribution of high-resolution remote sensing images can have complex characteristics (e.g., asymmetric or heavy-tailed), an innovative image segmentation algorithm is proposed based on the hierarchical Student’s-t mixture model (HSMM) and spatial constraints with adaptive smoothing. Considering the complex distribution of spectral intensities, the proposed algorithm constructs the HSMM to accurately build the statistical model of the image, making more reasonable use of the spectral information and improving segmentation accuracy. The component weight is defined by the attribute probability of neighborhood pixels to overcome the influence of image noise and make a simple and easy-to-implement structure. To avoid the effects of artificially setting the smoothing coefficient, the gradient optimization method is used to solve the model parameters, and the smoothing coefficient is optimized through iterations. The experimental results suggest that the proposed HSMM can accurately model asymmetric, heavy-tailed, and bimodal distributions. Compared with traditional segmentation algorithms, the proposed algorithm can effectively overcome noise and generate more accurate segmentation results for high-resolution remote sensing images.