This study presented a method for modeling the nonlinear system of a planetary gearbox and the fault diagnosis of a crack in a planetary gear based on the Volterra series theory. First, the exponential Hilbert reproducing kernel and its fast optimization algorithm was proposed and deduced in theory, and the fast solution of the fourth-order kernel of the Volterra series was successfully solved. Second, the Volterra series model estimation was compared with the least squares estimation of the actual collected signals from the planetary gearbox and the time-domain output signal was estimated using a neural network. The accuracy and the superiority of the Volterra series model of the planetary gearbox were then verified. At the same time, the convergence and the memory length of the Volterra series were discussed. In order to further mine and extract fault feature information, coupling relationship between the generalized frequency response of higher order spectrum of the Volterra series model and fault frequency was also studied. This study attempted to reflect the fault state and fault degree of a crack in a planetary gear from different observation angles and dimensions. Finally, the real condition loading test of a gearbox's comprehensive fault test platform was carried out. The validity of the method of nonlinear system modeling and fault diagnosis of the planetary gearbox, based on the Volterra series theory, was verified, and a new solution has been provided for related research in this field.