Mangrove ecosystems play an irreplaceable role in coastal environments by providing essential ecosystem services. Diverse mangrove species have different functions due to their morphological and physiological characteristics. A precise spatial distribution map of mangrove species is therefore crucial for biodiversity maintenance and environmental conservation of coastal ecosystems. Traditional satellite data are limited in fine-scale mangrove species classification due to low spatial resolution and less spectral information. This study employed unmanned aerial vehicle (UAV) technology to acquire high-resolution multispectral and hyperspectral mangrove forest imagery in Guangxi, China. We leveraged advanced algorithms, including RFE-RF for feature selection and machine learning models (Adaptive Boosting (AdaBoost), eXtreme Gradient Boosting (XGBoost), Random Forest (RF), and Light Gradient Boosting Machine (LightGBM)), to achieve mangrove species mapping with high classification accuracy. The study assessed the classification performance of these four machine learning models for two types of image data (UAV multispectral and hyperspectral imagery), respectively. The results demonstrated that hyperspectral imagery had superiority over multispectral data by offering enhanced noise reduction and classification performance. Hyperspectral imagery produced mangrove species classification with overall accuracy (OA) higher than 91% across the four machine learning models. LightGBM achieved the highest OA of 97.15% and kappa coefficient (Kappa) of 0.97 based on hyperspectral imagery. Dimensionality reduction and feature extraction techniques were effectively applied to the UAV data, with vegetation indices proving to be particularly valuable for species classification. The present research underscored the effectiveness of UAV hyperspectral images using machine learning models for fine-scale mangrove species classification. This approach has the potential to significantly improve ecological management and conservation strategies, providing a robust framework for monitoring and safeguarding these essential coastal habitats.