Rotor system supported by nonlinear bearing such as squeeze film damper (SFD) is widely used in practice owing to its wide range of damping capacity and simplicity in structure. In this paper, an improved and effective Incremental transfer matrix method (ITMM) is first presented by combining ITMM and fast Fourier transform (FFT). Afterwards this method is applied to calculate the dynamic characteristics of a Jeffcott rotor system with SFD. The convergence difficulties incurred caused by strong nonlinearities of SFD has been dealt by adopting a control factor. It is found that for the more general boundary problems where the boundary conditions are not at input and output ends of a chain system, the supplementary equation is necessarily added. Additionally, the Floquet theory is used to analyze the stability and bifurcation type of the obtained periodic solution. The semi-analytical results, including the periodic solutions of the system, the bifurcation points and their types, are in good agreement with the numerical method. Furthermore, the involution mechanism of the quasi-periodic and chaotic motions near the first-order translational mode and the second order bending mode of this system is also clarified by this method with the aid of Floquet theory. Keywords Incremental transfer matrix method • Squeeze film damper • Jeffcott rotor system • Bifurcation • Nonlinear characteristics Abbreviations a, a si , a ci Fourier series, dimensionless , Displacement and angular components in the state vector c Damping coefficient of linear elastic support, N ⋅ s/m C Clearance of squeeze film damper, m