A nonsynchronous low frequency vibration occurs in high-speed motor system with active magnetic bearing (AMB) was studied. The mechanism was analyzed from three aspects of actuator saturation, eddy current effects and external noises. The results show that the gain decrease caused by the actuator saturation is the main reason to induce the low frequency vibration, and the nonlinearity of electromagnetic force and switching power amplifier will further exacerbate the performance. The eddy current effects and external noises did not directly contribute to the low frequency vibration, but they will make the actuator more prone to saturation, so as to involve this matter. Then, the stability of AMB system with saturation nonlinearity was analyzed according to the extended Nyquist stability criterion method, and the compensation strategy based on the synchronous rotating frame (SRF) was present. For verification, a prototype of 100kW AMB-based PMSM and its system were developed and the experiments regarding to vibration response were carried out. The experimental results showed that the proposed compensation strategy could effectively filter out the synchronous component of the control current, so as to prevent the actuator saturation. In addition, during the rated speed of 30000r/m, the maximum rotor vibration amplitude with the compensator was reduced by about 21% compared with alone PID controller.