In order to study the stability of the wind turbine drivetrain in further depth, we present a nonlinear relative rotation mathematical model considering the nonlinear time-varying stiffness and the nonlinear damping force. Meanwhile, the nonlinear dynamics of the model under combined harmonic excitation are studied in detail. And some interesting dynamic phenomena are observed visually. Furthermore, to suppress chaotic oscillation within bounded time independent of initial conditions, a novel adaptive fixed-time terminal sliding mode controller is proposed. The stability of the final closed loop system is guaranteed according to Lyapunov stability theory. Rigorous mathematical analyses are used to prove the validity of the presented approach. Finally, compared with the existing finite-time stability method, simulation results are given to highlight the effectiveness and superiority of the proposed method and verify the theoretical analyses.