Motors are the main driving power for equipment operation, and they are also a major factor to promote the development of the motor and the load it drives and its motor control system toward a low-carbon future, reduce carbon emissions, and improve the industrial economy and social economic efficiency. Due to high-speed, long-period, and heavy-load operation, various faults occur; since the existing integer-order Fourier transform methods have not enough able to detect fractional-order faults and lack robustness, it is difficult to realize the fine diagnosis of motor faults, which reduces the safety and reliability of the motor control system. For this reason, on the basis of the powerful extraction ability of the fractional Fourier transform (FRFT) for micro fault features, especially the extraction ability to fit fractional frequency domain faults, this paper intends to establish a multilevel fine fault diagnosis method for fractional-order or integer-order faults. Firstly, this is accomplished by performing the fractional Fourier transform on the acquired data with faults and feature extraction in the multilevel fractional frequency domain and then optimizing the feature extraction model. Secondly, one further step search method is established to determine the projection direction with the largest fault feature. Thirdly, taking the extracted multilevel fault features as input, a multilevel fine fault diagnosis method based on the SVM model is established. Finally, three typical digital simulation examples and actual operating data collected by the ZHS-2 multifunctional motor test bench with a flexible rotor are employed to verify the effectiveness, robustness, and accuracy of this new method. The main contribution and innovation of this paper are that the fractional Fourier transform method based on time domain and frequency domains is introduced. This method can extract the small fault features in the maximum projection direction of the signal in the fractional domain, but detection with other time–frequency methods is difficult; the extracted multilevel fault features are used as input, and the corresponding fault diagnosis model is established, which can improve the accuracy of fault detection and ensure the safe and reliable operation of industrial equipment.