Due to continuous degradation of mangrove forests, the accurate monitoring of spatial distribution and species composition of mangroves is essential for restoration, conservation and management of coastal ecosystems. With leaf hyperspectral reflectance, this study aimed to explore the potential of continuous wavelet analysis (CWA) combined with different sample subset partition (stratified random sampling (STRAT), Kennard-Stone sampling algorithm (KS), and sample subset partition based on joint X-Y distances (SPXY)) and feature extraction methods (principal component analysis (PCA), successive projections algorithm (SPA), and vegetation index (VI)) in mangrove species classification. A total of 301 mangrove leaf samples with four species (Avicennia marina, Bruguiera gymnorrhiza, Kandelia obovate and Aegiceras corniculatum) were collected across six different regions. The smoothed reflectance (Smth) and first derivative reflectance (Der) spectra were subjected to CWA using different wavelet scales, and a total of 270 random forest classification models were established and compared. Among the 120 models with CWA of Smth, 88.3% of models increased the overall accuracy (OA) values with an improvement of 0.2–28.6% compared to the model with the Smth spectra; among the 120 models with CWA of Der, 25.8% of models increased the OA values with an improvement of 0.1–11.4% compared to the model with the Der spectra. The model with CWA of Der at the scale of 23 coupling with STRAT and SPA achieved the best classification result (OA = 98.0%), while the best model with Smth and Der alone had OA values of 86.3% and 93.0%, respectively. Moreover, the models using STRAT outperformed those using KS and SPXY, and the models using PCA and SPA had better performances than those using VIs. We have concluded that CWA with suitable scales holds great potential in improving the classification accuracy of mangrove species, and that STRAT combined with the PCA or SPA method is also recommended to improve classification performance. These results may lay the foundation for further studies with UAV-acquired or satellite hyperspectral data, and the encouraging performance of CWA for mangrove species classification can also be extended to other plant species.