With the increasing proportion of renewable power generations in the power system, the power grid impedance may fluctuate greatly, and it is difficult for the virtual synchronous generator (VSG) with a single control structure to meet the stability requirements. Thus, the control principles of the two types of VSGs are compared to conclude that there are similarities in the control structure of the two VSGs. And the small signal models of the two types of VSGs are established to analyze the stability boundaries, coming to the conclusion that the stability region is complementary. Based on the conclusions, the paper puts forward the idea of switching operation modes: switch to the U‐VSG mode when the grid strength weakens and switch to the PQ‐VSG mode when greater, so that the inverter can keep stable in a wider range of grid strength. Therefore, the characterization method of the switching boundary with hysteresis properties is proposed. Then the improved recursive least squares (RLS) algorithm is introduced to identify the power grid impedance online without additional injected disturbance, based on which the adaptive multi‐mode smooth switching control strategy is proposed. Finally, the effectiveness of the analysis and the control strategy are verified by simulations.