As the core component of the wind turbine generation gearbox, the gear-rotor-bearing transmission system typically operates in harsh environments, inevitably leading to the occurrence of composite faults in the system, which exacerbates system vibration. Therefore, it is necessary to study the vibration characteristics of wind turbine helical gear-rotor-bearing transmission systems with composite faults. This paper uses an improved energy method to calculate the theoretical time-varying mesh stiffness of a helical gear with a root crack failure. On the premise of considering the time-varying meshing stiffness of the faulty helical gear, the gear eccentric fault, and the nonlinear support force of the faulty bearing, a multi-degree-of-freedom helical gear-rotor-bearing transmission system with compound faults was established by using the lumped parameter method. The dynamic model of the system was solved based on the Runge–Kutta method, and the vibration response of the system under healthy conditions, single faults with gear eccentricity, single faults with tooth root cracks, and coupled bearing composite faults were simulated and analyzed. The results show that the simulation results based on KISSsoft software 2018 version verify the effectiveness of the improved energy method; the existence of single faults and composite faults will cause the fault characteristics in the time domain and frequency domain responses. In this paper, the influence of a single fault and a complex fault on the time domain and frequency domain of the system is mainly discovered through the fault study of the helical rotor-bearing system, and the influence of the fault degree on the vibration of the gear motion system is discussed. The greater the degree of the fault, the more vibration of the system occurs; accordingly, when the system is under the coupling of tooth root crack and bearing fault, there is a significant difference compared with the healthy system and the single fault system. The system vibration has obvious time domain and frequency domain signal characteristics, including periodic pulse impacts caused by gear faults and time domain impact caused by bearing. The fault characteristic frequencies can also be found in the frequency domain. In this paper, the fault study of a helical gear of wind turbine generation provides a reference for the theoretical analysis of the vibration characteristics of the helical gear-rotor-bearing system under various fault conditions, lays a solid foundation for the simulation and subsequent diagnosis of the composite fault signal of the system, and provides help for the fault diagnosis of wind turbine gearboxes in the future.