In order to ensure the stability and reliability of power supply and realize day and night power generation, wind and solar complementary power generation systems are built in areas with abundant solar and wind energy resources. However, the system investment cost is too high. Because of this, there are wind, light intermittent, and non-intermittent power generation systems. For issues such as stability, an energy storage system needs to be configured to stabilize power fluctuations. This paper aims to study the optimization control of hybrid energy storage system of new energy power generation system based on improved particle swarm algorithm. In this paper, the application of particle swarm algorithm to power system reactive power optimization has been researched in two aspects. Through optimization methods, reasonable adjustment of control variables, full use of equipment resources of the power grid, to improve voltage quality and reduce system operation network to ensure the stability of the voltage system. In addition, this paper selects the IEEE30 node test system and simulation data analysis, takes the hybrid energy storage system as the optimization object, and optimizes the reactive power of the newly improved particle swarm algorithm. The experiments in this paper show that the improved algorithm has a good effect in reactive power optimization, increasing the performance of the hybrid energy storage system by 27.02%. MPSO algorithm is also better than basic PSO algorithm. It can be seen from the figure that in the PSO algorithm, the algorithm basically tends to be stable after more than 40 iterations, and finally the algorithm converges to 0.089.