The penetration of distributed generators (DGs)-based power electronic devices leads to low inertia and damping properties of the modern power grid. As a result, the system becomes more susceptible to disruption and instability, particularly when the power demand changes during critical loads or the system needs to switch from standalone to a grid-connected operation mode or vice versa. Developing a robust controller to deal with these transient cases is a real challenge. The inverter control method via the virtual synchronous generator (VSG) control method is a better way to supply the system’s inertia and damping features to boost system stability. Therefore, a nonlinear control strategy for VSG with uncertain disturbance is proposed in this paper to enhance the system stability in the islanded, grid-connected, and transition modes. Firstly, the mechanical equations for a VSG’s rotor, which include virtual inertia and damping coefficient, are presented, and the matching mathematical model is produced. Then, the nonlinear backstepping controller (BSC) method combined with the extended state observer (ESO) is constructed to compensate for the uncertainty. The Lyapunov criteria were used to prove the method’s stability. Considering the issue of uncertain items, a second-order ESO is built to estimate uncertainty and external disruption. Finally, the suggested control strategy is validated through three simulation experiments; the findings reveal that the proposed control method has an excellent performance with fast response and tracking under various operating situations.