A double modulation phenomenon is pointed out in the wind turbine gearbox compound fault signals, which consists of a resonance modulation frequency band and an asymmetric modulation frequency band. The modal aliasing and the double modulation phenomena make it difficult for empirical wavelet transform to obtain the spectral components in the meshing modulation regions. The main contribution is to establish an optimal empirical wavelet transform (OEWT) framework for weak feature extraction of compound fault, which incorporates optimal velocity synchronous linear chirplet transform (OVSLCT) and simulated annealing (SA) algorithm to obtain the optimal parameters of the filter bank. The frequency boundary of the fault component is obtained through OVSLCT rather than the extreme value of the spectrum. A filter transition band width optimization scheme is proposed, in which the optimal transition band width of the filter bank in EWT is optimized by SA. Compared with the original EWT and VMD, OEWT can better obtain the compound fault characteristic frequency and solve the mode aliasing problem.