Permanent magnet synchronous motor (PMSM) is a vital component of modern industry which is widely used in transport, aerospace and intelligent machinery. In the light of its relatively high frequency of inter-turn short circuit (ITSC) fault, it is valuable to detect the fault accurately and efficiently. Due to the convenience of motor vibration signal measurement, more attention has been paid to fault detection methods based on vibration signal. As a nonstationary, nonlinear signal, vibration signal is highly susceptible to external noise interference. Variational mode decomposition (VMD) has a wide range of applications in the field of nonlinear signal analysis, but the processing ability to the signal is affected by the parameter setting. Hence, a Bayesian optimized adaptive parameter selection variational mode decomposition (BOAPS-VMD) signal processing method is proposed and applied to detect the ITSC in PMSM. Firstly, the motor vibration signal is decomposed through BOAPS-VMD, and the intrinsic mode functions (IMFs) are obtained. Secondly, the cumulative variance contribution rate (C-VCR) is applied to identify the IMFs that contain fault signature information. Finally, using Hilbert transform (HT) to further analyze the IMFs that are identified. The results are output in 3D time-frequency diagrams to enhance the representation of fault signatures. The effectiveness and accuracy of the BOAPS-VMD were validated through the finite element simulation and experiment. The study shows that the BOAPS-VMD effectively improves the modal mixing phenomenon, has better noise robustness, improves the accuracy of fault detection, and has better engineering applicability.INDEX TERMS Permanent magnet synchronous motor (PMSM), fault detection, variational mode decomposition (VMD), Bayesian optimization, Hilbert transform (HT)