The vibration signal of rolling bearing fault is nonlinear and nonstationary under the interference of background noise, and it is difficult to extract fault features from it. When feature mode decomposition is used to analyze signals, prior parameter settings can easily affect the decomposition results. Therefore, a fault feature extraction method based on improved whale optimization algorithm is proposed to optimize feature modal decomposition parameters. The improved WOA integrates Lévy flight and adaptive weight, and envelope entropy is used as fitness function to optimize feature modal decomposition parameters. The feature mode decomposition of the original signal is performed using the optimal combination of parameters to obtain multiple IMF components. The optimal IMF component envelope demodulation analysis is selected according to the kurtosis value, and the fault feature is extracted through the envelope spectrum. Comparing the LMWOA method with PSO and WOA methods by simulated and experimental signals, the results show that the optimization speed of LMWOA is faster than that of other methods. Compared with CEEMD, VMD, and FMD methods, the improved WOA-FMD method has higher fault feature ratio and can accurately extract fault features under noise interference. This method can effectively solve the parameter adaptive ability and improve the accuracy of fault diagnosis, which has practical significance.