The torsional dynamic model of double-helical gear pair considering time-varying meshing stiffness, constant backlash, dynamic backlash, static transmission error, and external dynamic excitation was established. The frequency response characteristics of the system under constant and dynamic backlashes were solved by the incremental harmonic balance method, and the results were further verified by the numerical integration method. At the same time, the influence of time-varying meshing stiffness, damping, static transmission error, and external load excitation on the amplitude frequency characteristics of the system was analyzed. The results show that there is not only main harmonic response but also superharmonic response in the system. The time-varying meshing stiffness and static transmission error can stimulate the amplitude frequency response of the system, while the damping can restrain the amplitude frequency response of the system. Changing the external load excitation has little effect on the amplitude frequency response state change of the system. Compared with the constant backlash, increasing the dynamic backlash amplitude can further control the nonlinear vibration of the gear system.