Squeeze film damper (SFD) is widely used in the vibration suppression of aeroengine rotor systems, but will cause complex motions of the rotor system under specific operating conditions. In this paper, a lumped-mass dynamic model of the high-pressure rotor system in an aeroengine is established, and the nonlinear stiffness and damping formula of SFD are introduced into the above model. The vibration responses of the rotor system under different rotating speeds and with different unbalances are investigated numerically, and the influence of SFD on the rotor system vibration and the change of suppression ability are compared and analyzed. The results show that in the case of high speed, together with a small unbalance, the rotor system will perform a complex vibration or a bistable vibration due to SFD. If the unbalance is properly increased under the same case of high speed, the vibration of the rotor becomes single-harmonic and the bistable vibration disappears. The research results can provide a helpful reference for analyzing complex vibration mechanisms of the rotor system with SFD and achieving an effective vibration suppression through unbalance regulation.