With the increasing service life of a gear transmission system, the gear teeth become constantly worn, and the gear clearance increases. The increase in the clearance will produce a series of nonlinear changes that change the stability of the system and can even result in loss of stability. In this paper, dimensionless dynamic equations of a multistage gear transmission system that contains a two-stage fixed-axis gear and a one-stage planetary gear were established. Planetary wear fault was simulated by changes in the gear clearance. System bifurcation diagrams with an increase in the clearance were studied. The frequency characteristics of planetary gears under different excitation frequencies and different degrees of wear were studied. The influence of planetary gear wear on the fixed-axis was discovered. The vibration mechanism and a fault diagnostic method in a multistage gear transmission system were obtained.