In order to study the vibration characteristics and fault characteristics of double-ring herringbone planetary gear transmission system under different gear tooth broken faults. The 3D model of the system is established using SolidWorks, and based on ADAMS virtual prototype and Hertz contact theory, considering the friction coefficient. The angular acceleration spectrum characteristics of the system under the condition of health and fault are studied. The results show that the fault characteristics of the system are obvious when the gear tooth broken fault occurs. Along with the tooth broken failure to deepen, the side-band frequencies on both sides of the fundamental frequency and harmonic frequency of the system are gradually dense. The sun gear has obvious response to the system asymmetry caused by the gear tooth broken fault. The results of the planetary gear system fault diagnosis research and design has a certain reference value.