Mangroves are one of the most important coastal wetland ecosystems, and the compositions and distributions of mangrove species are essential for conservation and restoration efforts. Many studies have explored this topic using remote sensing images that were obtained by satellite-borne and airborne sensors, which are known to be efficient for monitoring the mangrove ecosystem. With improvements in carrier platforms and sensor technology, unmanned aerial vehicles (UAVs) with high-resolution hyperspectral images in both spectral and spatial domains have been used to monitor crops, forests, and other landscapes of interest. This study aims to classify mangrove species on Qi'ao Island using object-based image analysis techniques based on UAV hyperspectral images obtained from a commercial hyperspectral imaging sensor (UHD 185) onboard a UAV platform. First, the image objects were obtained by segmenting the UAV hyperspectral image and the UAV-derived digital surface model (DSM) data. Second, spectral features, textural features, and vegetation indices (VIs) were extracted from the UAV hyperspectral image, and the UAV-derived DSM data were used to extract height information. Third, the classification and regression tree (CART) method was used to selection bands, and the correlation-based feature selection (CFS) algorithm was employed for feature reduction. Finally, the objects were classified into different mangrove species and other land covers based on their spectral and spatial characteristic differences. The classification results showed that when considering the three features (spectral features, textural features, and hyperspectral VIs), the overall classification accuracies of the two classifiers used in this paper, i.e., k-nearest neighbor (KNN) and support vector machine (SVM), were 76.12% (Kappa = 0.73) and 82.39% (Kappa = 0.801), respectively. After incorporating tree height into the classification features, the accuracy of species classification increased, and the overall classification accuracies of KNN and SVM reached 82.09% (Kappa = 0.797) and 88.66% (Kappa = 0.871), respectively. It is clear that SVM outperformed KNN for mangrove species classification. These results also suggest that height information is effective for discriminating mangrove species with similar spectral signatures, but different heights. In addition, the classification accuracy and performance of SVM can be further improved by feature reduction. The overall results provided evidence for the effectiveness and potential of UAV hyperspectral data for mangrove species identification.