Virtual synchronous generator (VSG) control is an attractive method to solve the stability issues of future renewables (e.g. photovoltaic [PV])‐dominated power system. Unfortunately, a VSG is prone to transient instability under grid faults, and the existing work ignores the instability issues caused by current limitation and reactive power loop (RPL). To tackle with the problem, this article analyzes the transient stability of VSG considering the impact of current limitation and RPL. It indicates that the roots of transient instability can be attributed to two aspects, that is, (1) the positive feedback and continuous phase angle integration in active power loop (APL) which are caused by the P‐f control structure, (2) the decline of active power transfer limit which is caused by the Qvsg‐vcd‐iod‐Pvsg control structure. Based on it, an optimized VSG control method is proposed to improve the transient stability, by optimizing the design of power loops for VSG. Compared with other stability control methods, the proposed method avoids the use of frequency detection, and it is simple in design and implementation. Finally, the proposed method is validated by the experiments.

No abstract