In order to identify the nonlinear nonstationary pitting-wear fault signal of gears in gearbox, a new method of composite fault diagnosis for gearbox is proposed, which combines empirical mode decomposition with improved variational mode decomposition. Aiming at the problem of false modes in the results of empirical mode decomposition when processing signals, the energy method is used to eliminate and update the mode components, and the correlation coefficient method is used to calculate the correlation between the updated mode components and the original signal. The components with strong correlation are selected to form the combined mode components to weaken the noise and improve the signal-to-noise ratio. Aiming at the problem that variational mode decomposition method needs to manually determine the number of mode components k and the penalty factor a during signal decomposition, a combination of envelope spectral entropy and waveform method is proposed to determine the optimal parameter combination. By analyzing the pitting-wear composite fault vibration signals of gears in the gearbox and the normal signals of the gearbox, the effectiveness of the proposed method is verified, and a comparative analysis with the empirical mode decomposition method is performed to highlight the superiority of the proposed method.