As a key component to adjust the speed and torque, double-fed speed up gearbox plays a vital role in reliability and stability for the wind turbine system. Considering the base helix angle, normal pressure angle, position angle, rotation of carrier and the mesh of the ring gear and planet gear, a coupled dynamic model for high-power wind turbine gearbox transmission system, which consists of two helical planetary stages and one helical gear stage was established using the lumped parameter method in this paper. Numerical integration applying the explicit 4/5th order Runge-Kutta algorithm was used to solve the proposed model and the dynamic responses of transmission system were investigated. Results show that the modal frequencies for the system are not coincided with the gear mesh frequencies of the stages. Then, the dynamic mesh forces of the 1 st stage in different models are compared and analyzed in frequency domain.