Tooth fractures and cracks are common defects in gear transmission systems. Crack propagation paths of the spur gears are acquired using the fracture mechanics theory. Based on the simulated crack propagation path, the flexible body model of the cracked gear system is established in multibody dynamic software ADAMS. The vibration signals of cracked gear under different torques obtained from the simulation are verified by the experimental results. Some statistical indictors (root mean square and Kurtosis) are used to quantify the damage degree. In addition, the fault features of the gear system under crack and tooth fracture faults are analyzed by utilizing the instantaneous energy. The results indicate that the amplitude of the harmonics of the meshing frequency and the sideband frequencies increase gradually with the increase of torques and crack lengths. Especially, for the tooth fracture fault, its influence on Kurtosis value is particularly obvious. Meanwhile, according to the instantaneous energy, the tooth fracture fault will result in severe energy impacts and cause a sharp increase in the vibration amplitude.