The helical gear system is an important form of transmission in high-speed trains, and helical gear failure has a great impact on the transmission performance. To investigate the influence of wear parameters on the nonlinear dynamics of gear systems, the wear fault parameters of a bending-torsion-shaft coupling mode with six degrees of freedom are established. Using the variable step fourth-order Runge-Kutta numerical integration method, the gear dynamics model with fault parameters is analyzed to get the dynamic response of the helical gear system. The system excitation frequency, evolution of system periodic motion, quasi-periodic motion, and chaotic motion with variable fault parameters are analyzed qualitatively based on the results, including the system status judgment criteria of phase plane graph, Poincaré cross-section graph, bifurcation diagram, and RMS. The results show that wear fault affects the system differently at different frequencies. Finally, the correctness of the conclusion is verified through experiments, and the impact on the actual application process is analyzed.Published by Copernicus Publications.