The Global Ecosystem Dynamics Investigation (GEDI) system provides essential data for estimating forest canopy height on a global scale. However, factors such as complex topography and dense canopy can significantly reduce the accuracy of GEDI canopy height estimations. We selected the South Taihang region of Henan Province, China, as our study area and proposed an optimization framework to improve GEDI canopy height estimation accuracy. This framework includes correcting geolocation errors in GEDI footprints, screening and analyzing features that affect estimation errors, and combining two regression models with feature selection methods. Our findings reveal a geolocation error of 4 to 6 m in GEDI footprints at the orbital scale, along with an overestimation of GEDI canopy height in the South Taihang region. Relative height (RH), waveform characteristics, topographic features, and canopy cover significantly influenced the estimation error. Some studies have suggested that GEDI canopy height estimates for areas with high canopy cover lead to underestimation, However, our study found that accuracy increased with higher canopy cover in complex terrain and dense vegetation. The model’s performance improved significantly after incorporating the canopy cover parameter into the optimization model. Overall, the R2 of the best-optimized model was improved from 0.06 to 0.61, the RMSE was decreased from 8.73 m to 2.23 m, and the rRMSE decreased from 65% to 17%, resulting in an accuracy improvement of 74.45%. In general, this study reveals the factors affecting the accuracy of GEDI canopy height estimation in areas with complex terrain and dense vegetation cover, on the premise of minimizing GEDI geolocation errors. Employing the proposed optimization framework significantly enhanced the accuracy of GEDI canopy height estimates. This study also highlighted the crucial role of canopy cover in improving the precision of GEDI canopy height estimation, providing an effective approach for forest monitoring in such regions and vegetation conditions. Future studies should further improve the classification of tree species and expand the diversity of sample tree species to test the accuracy of canopy height estimated by GEDI in different forest structures, consider the distortion of optical remote sensing images caused by rugged terrain, and further mine the information in GEDI waveforms so as to enhance the applicability of the optimization framework in more diverse forest environments.