Tree heights are the principal variables for forest plantation inventory. The increasing availability of high-resolution three-dimensional (3D) point clouds derived from low-cost Unmanned Aerial Vehicle (UAV) and modern photogrammetry offers an opportunity to generate a Canopy Height Model (CHM) in the mountainous areas. In this paper, we assessed the capabilities of tree height estimation using UAV-based Structure-from-Motion (SfM) photogrammetry and Semi-Global Matching (SGM). The former is utilized to generate 3D geometry, while the latter is used to generate dense point clouds from UAV imagery. The two algorithms were coupled with a Radial Basis Function (RBF) neural network to acquire CHMs in mountainous areas. This study focused on the performance of Digital Terrain Model (DTM) interpolation over complex terrains. With the UAV-based image acquisition and image-derived point clouds, we constructed a 5 cm-resolution Digital Surface Model (DSM), which was assessed against 14 independent checkpoints measured by a Real-Time Kinematic Global Positioning System RTK GPS. Results showed that the Root Mean Square Errors (RMSEs) of horizontal and vertical accuracies are approximately 5 cm and 10 cm, respectively. Bare-earth Index (BEI) and Shadow Index (SI) were used to separate ground points from the image-derived point clouds. The RBF neural network coupled with the Difference of Gaussian (DoG) was exploited to provide a favorable generalization for the DTM from 3D ground points with noisy data. CHMs were generated using the height value in each pixel of the DSM and by subtracting the corresponding DTM value. Individual tree heights were estimated using local maxima algorithm under a contour-surround constraint. Two forest plantations in mountainous areas were selected to evaluate the accuracy of estimating tree heights, rather than field measurements. Results indicated that the proposed method can construct a highly accurate DTM and effectively remove nontreetop maxima. Furthermore, the proposed method has been confirmed to be acceptable for tree height estimation in mountainous areas given the strong linear correlation of the measured and estimated tree heights and the acceptable t-test values. Overall, the low-cost UAV-based photogrammetry and RBF neural network can yield a highly accurate DTM over mountainous terrain, thereby making them particularly suitable for rapid and cost-effective estimation of tree heights of forest plantation in mountainous areas.