To obtain a high-accuracy vegetation classification of high-resolution UAV images, in this paper, a multi-angle hyperspectral remote sensing system was built using a six-rotor UAV and a Cubert S185 frame hyperspectral sensor. The application of UAV-based multi-angle remote sensing in fine vegetation classification was studied by combining a bidirectional reflectance distribution function (BRDF) model for multi-angle remote sensing and object-oriented classification methods. This method can not only effectively reduce the classification phenomena that influence different objects with similar spectra, but also benefit the construction of a canopy-level BRDF. Then, the importance of the BRDF characteristic parameters are discussed in detail. The results show that the overall classification accuracy (OA) of the vertical observation reflectance based on BRDF extrapolation (BRDF_0°) (63.9%) was approximately 24% higher than that based on digital orthophoto maps (DOM) (39.8%), and kappa using BRDF_0° was 0.573, which was higher than that using DOM (0.301); a combination of the hot spot and dark spot features, as well as model features, improved the OA and kappa to around 77% and 0.720, respectively. The reflectance features near hot spots were more conducive to distinguishing maize, soybean, and weeds than features near dark spots; the classification results obtained by combining the observation principal plane (BRDF_PP) and on the cross-principal plane (BRDF_CP) features were best (OA = 89.2%, kappa = 0.870), and especially, this combination could improve the distinction among different leaf-shaped trees. BRDF_PP features performed better than BRDF_CP features. The observation angles in the backward reflection direction of the principal plane performed better than those in the forward direction. The observation angles associated with the zenith angles between −10° and −20° were most favorable for vegetation classification (solar position: zenith angle 28.86°, azimuth 169.07°) (OA was around 75%–80%, kappa was around 0.700–0.790); additionally, the most frequently selected bands in the classification included the blue band (466 nm–492 nm), green band (494 nm–570 nm), red band (642 nm–690 nm), red edge band (694 nm–774 nm), and the near-infrared band (810 nm–882 nm). Overall, the research results promote the application of multi-angle remote sensing technology in vegetation information extraction and provide important theoretical significance and application value for regional and global vegetation and ecological monitoring.