In this paper, relying on the Volterra series nonlinear system model and the high-order kernel Hilbert’s reconstructed kernel fast solved algorithm, a fault feature frequency domain identification method based on Volterra high-order kernel generalized frequency response graph analysis is proposed. Firstly, the method uses the system input and output vibration signals to determine the Volterra model. Then, the Volterra high-order kernel function is solved quickly by reproducing kernel Hilbert space method, and the generalized frequency response function is used to identify the model. Finally, multidimensional high-order spectral pattern analysis is used to separate and extract the fault and degree characteristic information implied by frequency and phase coupling in the third-order kernel function. Following the theoretical approach, in the experimental part, this paper uses the planetary gearbox fault loading test rig to complete the data collection and establishes the Volterra experimental model through the measured data. The generalized frequency responses of each order kernel function are compared and analyzed and the capability of distinguishing and the adaptability of different order kernel functions for the degree of crack failure are discussed. The effects of changing the memory length of the Volterra model and the order of the kernel function on the recognition result are verified. The final experimental results show that the use of reproducing kernel Hilbert space can effectively avoid the dimension disaster problem that occurs in the high-order kernel solution process. Moreover, the third-order kernel can describe more intuitively the nonlinear system model under multifactor coupling than the second-order kernel. Finally, Volterra series model the third-order kernel’s generalized frequency response can effectively distinguish between nondefective and faulty gears, and its resolution is enough to distinguish the degree of failure of gear cracks.