Forest height inversion with Polarimetric SAR Interferometry (PolInSAR) has become a research hotspot in the field of radar remote sensing. In this paper, we systematically studied a modified two-step, three-stage inversion simulating the L-band (L = 23 cm) full-polarization interferometric SAR data with an average forest height of 18 m using ESA PolSARpro-SIM software. We applied this method to E-SAR L-band single-baseline full PolInSAR data in 2003. In the first step, we modified the three-stage inversion algorithm based on phase diversity (PD)/maximum coherence difference (MCD) coherence optimization methods, corresponding to PD, MCD, respectively. In the second step, we introduced the coherence amplitude inversion term and modified the fixed weight to the variable of ε times the ground scattering ratio, which improved the accuracy of forest height inversion. The mean of forest height inversion by the HV method was the lowest (15.83 m) and the RMSE was the largest (4.80 m). The PD method was superior to the HV method with RMSE (4.60 m). The MCD method was slightly better than using the PD method with the smallest RMSE (4.43 m). After adding the coherence amplitude term, the RMSE was improved by 0.15 m, 0.14 m, and 0.08 m, respectively. The smallest RMSE was obtained by MCD, followed by the PD and HV methods. Although the robustness of the forest height inversion algorithm was reduced, the underestimation was improved and the RMSE was reduced. Due to the complexity of the real SAR E-SAR L-band single-baseline full PolInSAR data and the small sample sizes, the three-stage inversion methods based on coherent optimization were lower than the three-stage in-version method. After introducing the coherent magnitude term, the overestimation of the forest height was significantly weakened in HVWeight, PDweight, and MCDWeight, and PDWeight was optimal. The modified two-step, three-stage inversion algorithm had significant effects in alleviating forest height underestimation and overestimation, improving the accuracy of forest height inversion, and laying a foundation for the upcoming L-band SAR satellite generation, new SAR and LIDAR systems combined with RPAs (remotely piloted aircrafts)/UAVs (unmanned aerial vehicles) for small areas mapping initiatives, and promoting the depth and breadth of the SAR applications of the new SAR system.