As an important parameter of forest biomass, forest height is of great significance for the calculation of forest carbon stock and the study of the carbon cycle in large-scale regions. The main idea of the current forest height inversion methods using multi-baseline P-band polarimetric interferometric synthetic aperture radar (PolInSAR) data is to select the best baseline for forest height inversion. However, the approach of selecting the optimal baseline for forest height inversion results in the process of forest height inversion being unable to fully utilize the abundant observation data. In this paper, to solve the problem, we propose a multi-baseline forest height inversion method combining analytic and geometric expression of the random volume over ground (RVoG) model, which takes into account the advantages of the selection of the optimal observation baseline and the utilization of multi-baseline information. In this approach, for any related pixel, an optimal baseline is selected according to the geometric structure of the coherence region shape and the functional model for forest height inversion is established by the RVoG model’s analytic expression. In this way, the other baseline observations are transformed into a constraint condition according to the RVoG model’s geometric expression and are also involved in the forest height inversion. PolInSAR data were used to validate the proposed multi-baseline forest height inversion method. The results show that the accuracy of the forest height inversion with the algorithm proposed in this paper in a coniferous forest area and tropical rainforest area was improved by 17% and 39%, respectively. The method proposed in this paper provides a multi-baseline PolInSAR forest height inversion scheme for exploring regional high-precision forest height distribution. The scheme is an applicable method for large-scale, high-precision forest height inversion tasks.