Accurate and efficient detection of canopy gaps is essential for understanding species regeneration and community dynamics in forests. Unoccupied aerial vehicles (UAVs) equipped with visible light (e.g., RGB) cameras have the potential to be one of the most cost‐effective approaches for detecting gaps. However, current gap‐detection methods based on spectral, textural, and/or structural information derived from UAV RGB imagery are unreliable in species‐rich forests with complex terrain due to high spectral complexity and topographic shadowing. Here, we compared the performance of four methods, including pixel‐based supervised classification (PBSC), object‐based classification (OBIA), Canopy Height Model thresholding classification, and HSTAC [a novel method we developed which combines Photographic Height (H), Spectral (S), and Textural (T) information for Automatic Classification (AC)] for characterizing canopy gaps in a 20‐ha permanent subtropical forest plot of eastern China. All classification results were evaluated through a comparison with canopy gaps detected from both field surveys and UAV‐borne LiDAR data. Among the four classification methods, HSTAC performed best in terms of detection efficiency (96% overall accuracy when compared to field data and 85% when compared to the LiDAR data), classification accuracy (3–18% improvement compared to alternative methods), and speed (1–1.5 h faster on the same machine). Of the four topographic factors (elevation, slope, aspect, and convexity), elevation was the one that most affected the accuracy of canopy gap detection. The errors of PBSC classification mainly came from the gaps at low elevations, while OBIA located the position of gaps well but overestimated their sizes. Overall, HSTAC avoids many of the inherent limitations of current state‐of‐the‐art methods and can accurately map canopy gaps in diverse subtropical forests with complex terrain. Our study provides a suitable way for long‐term forest canopy monitoring, real‐time applications, and contributes to a better understanding of forest plant community assembly and succession dynamics.