The research described in this study is aimed at exploring the transmission characteristics of a two-stage planetary-gear transmission system. Firstly, a novel rigid-flexible coupled dynamic model was established with the consideration of gear clearance, time-varying meshing stiffness, and the deformation of the bearing and shaft. Secondly, based on the novel dynamic model, different profile modification parameters are carried out to optimize the transmission error, load distributions, and harmonic quantity of the transmission system. The analysis results show that dynamic characteristics can be improved by optimizing the multi-stage planetary gear profile. The study provides a valuable reference for the design of multi-stage planetary-gear transmission systems of hoist industry.