Aiming at the problem of pitch error of helical gear pair in engineering practice, the influence of pitch error on vibration, bifurcation, and chaos characteristics of the helical gear pair system is mainly studied. Due to the periodic time-varying nature of pitch error, a method of simulating the pitch error as a sine function is proposed to calculate pitch error. A nonlinear dynamic model of bending-torsion-shaft coupling of the helical gear pair system is established considering the effect of pitch error. The influence of pitch error on the vibration, bifurcation, and chaos characteristics of the system is analyzed by the Runge–Kutta numerical integration method. The research results show that the introduction of pitch error has the most significant impact on the torsional vibration of the system. With the increase in pitch error, the system exhibits rich bifurcation and chaos characteristics in the torsional direction. Moreover, it is also found that the vibration response in the torsional orientation of the system increases or decreases to the same degree when the system is in a periodic motion state, and the pitch error varies by the same extent. Therefore, the impact of pitch error on the dynamic performance of the helical gear pair system should be considered in engineering practice.