The oscillating flow is an important factor affecting the performance of the rotor–seal system. From the point of view of flow induced vibration, the nonlinear models of the rotor-seal system are presented for the analysis of the self-excited vibration, which is induced by interaction between the unstable seal fluid flow and the vibrating rotor. The nonlinear characteristics of flow induced vibration in the rotor-seal system are analyzed, and the nonlinear phenomena in the unbalanced rotor-seal system are investigated using the nonlinear model with the flow induced vibration.