Tooth spalling is one of the most destructive surface failure models of the gear faults. Previous studies have mainly concentrated on the spalling damage of a single gear tooth, but the spalling distributed over double teeth, which usually occurs in practical engineering problems, is rarely reported. To remedy this deficiency, this paper constructs a new dynamical model of a gear system with double-teeth spalling fault and validates this model with various experimental tests. The dynamic characteristics of gear systems are obtained by considering the excitations induced by the number of spalling teeth, and the relative position of two faulty teeth. Moreover, to ensure the accuracy of dynamic model verification results and reduce the difficulty of fault feature analysis, a novel parameter-adaptive variational mode decomposition (VMD) method based on the ant lion optimization (ALO) is proposed to eliminate the background noise from the experimental signal. First, the ALO is used for the self-selection of the decomposition number K and the penalty factor â of the VMD. Then, the raw signal is decomposed into a set of Intrinsic Mode Functions (IMFs) by applying the ALO-VMD, and the IMFs whose effective weight kurtosis (EWK) is greater than zero are selected as the reconstructed signal. Combined with envelope spectrum analysis, the de-nosing ability of the proposed method is compared with that of the method known as particle swarm optimization-based variational mode decomposition (PSO-VMD), the fixed-parameter VMD, the empirical mode decomposition (EMD), and the local mean decomposition (LMD), respectively. The results indicate that the proposed dynamic model and background elimination method can provide a theoretical basis for spalling defect diagnosis of gear systems.