Rotor-Active magnetic bearing (rotor-AMB) systems offer significant advantages in power consumption and are increasingly being applied in fluid machinery. However, in such machinery, the impeller and the rotor are commonly connected by bolt fastening. Once a certain pre-tightening force is applied to the bolted connection and the interface contact between the rotor and the impeller is formed, the flexible modal vibration will be excited when the rotor is levitated. The conventional Proportional-Integral-Derivative (PID) controller is limited in this vibration suppression and it is essential to increase the damping at bending mode to inhibit vibration. However, the calculation of the optimal phase compensated is based on the rotor-AMB model considering interface contact. In this work, the effect of the interface contact is equivalent to massless spring units, and the disturbance generated by spring units is acted upon the mechatronic system in the way of positive feedback. The detailed description of the transfer function from the disturbance to rotor deformation at fourth bending mode is obtained by theoretical model and system identification. The optimal phase of the system needed to inhibit the disturbance is determined by minimizing the gain of the transfer function at fourth bending mode. A combination of PID control and phase compensation is implemented in the experiment. The results show that the optimal phase is 50° as the calculation and the optimal phase compensation control can sufficiently inhibit the modal vibration.