Forest canopy height is an important forest indicator parameter. Synthetic aperture radar tomography (TomoSAR) is an effective method to characterize forest canopy height and describe forest 3D structure; however, the residual phase error of TomoSAR affects the focus of the relative reflectance and can lead to errors in forest canopy height estimation. Therefore, this paper proposes a semi-empirical method to overcome the residual phase effects on forest canopy height estimation. In this study, we used airborne multi-baseline UAVSAR data to estimate forest canopy height via TomoSAR techniques and applied a semi-empirical method to improve forest canopy height estimation without phase calibration to mitigate the effects of phase error. The process is divided into three stages: the first step uses a semi-empirical method to initially determine the optimal relative reflectance loss threshold (K) by excluding the inverse extremes; in the second and third steps, the percentile height was used to gradually reduce the height interval between the upper and lower envelopes to minimize overestimation of extreme values and the lower vegetation. When the root mean square error (RMSE) was minimized, the percentile combinations were determined between the inversion results and a LiDAR dataset of the area. The results show that the canopy height estimation results are not satisfactory when relying solely on the K value to estimate the height difference between the envelope at the top of the forest and the ground; the best result was obtained when K = 0.4, but the corresponding R2 value was only 0.13, and the RMSE was 15.23 m. In our proposed method, the K value is determined as 0.3 by excluding the extreme values of the inversion result in the initial step—the corresponding R2 and RMSE values were 0.59 and 10.73 m, respectively, representing an RMSE decrease of 29.54% relative to the initial K value. After two steps of correction overestimation, the inversion accuracy was significantly improved with an R2 value of 0.65 and an RMSE of 9.69 m, corresponding to an RMSE decrease of 36.38%. Overall, the findings of the study represent an important reference for optimizing future spaceborne TomoSAR forest canopy height estimates.