To analyze the vibration characteristic differences of a gear transmission system with gear and shaft cracks, an improved computational method for deriving the shaft stiffness matrix with breathing cracks is developed. Also, the three-dimensional contact model of spur gear with a crack is established via the finite element method (FEM), and its meshing stiffness is calculated. Simultaneously, considering bearing stiffness and shaft flexibility, the finite element dynamic models of two-stage gear transmission system with gear and shaft cracks are established. Based on this, different source fault vibration responses are compared and the influencing factors are explored. In addition, a novel signal processing method based on the particle swarm optimization, maximum correlated kurtosis deconvolution, variational, mode decomposition and fast spectral kurtosis (PSO-MCKD-VMD-FSK) is utilized to extract fault characteristics for the signal-to-noise ratio and uneven energy distribution problems. Results show that a system with gear cracks mainly presents periodic impact in the time domain, while in the frequency domain it impacts rotation frequency modulation near the meshing frequency and its multiple frequencies. However, the shaft crack breathing effect meant that the time domain mainly presents "simple harmonic" modulation, and the rotation frequency and its faulty shaft multiplication occurs in the low-frequency region of the frequency domain. The PSO-MCKD-VMD-FSK method extracts fault features in a strong noise environment and has good robustness. Results identify different source faults and provide a theoretical basis.