The bifurcation and Lyapunov exponent for a single-machine-infinite bus system with excitation model are carried out by varying the mechanical power, generator damping factor and the exciter gain, from which periodic motions, chaos and the divergence of system are observed respectively. From given parameters and different initial conditions, the coexisting motions are developed in power system. The dynamic behaviors in power system may switch freely between the coexisting motions, which will bring huge security menace to protection operation. Especially, the angle divergences due to the break of stable chaotic oscillation are found which causes the instability of power system. Finally, a new adaptive backstepping sliding mode controller is designed which aims to eliminate the angle divergences and make the power system run in stable orbits. Numerical simulations are illustrated to verify the effectivity of the proposed method.