The coupled heave and pitch motions of a ship sailing in head waves affect the stability of the marine rotor-bearing system. Based on the theory of analytical mechanics, this study establishes a dynamic model of the rotor-bearing system subjected to the coupled motions of heave and pitch, considering nonlinear oil film moments produced by the tilting of the rotor in the bearings. The nonlinear dynamic behaviours of the system are analysed using numerical methods to obtain Poincaré sections, bifurcation diagrams, and the largest Lyapunov exponents. The results show that dynamic bifurcation characteristics reveal complex quasi-periodic motion of upper and lower branches after the initial instability of the system, and the speed of second instability increase markedly. At high speeds, the amplitude of the rotor system increases sharply, which can cause the rotor to touch the inner wall of the bearings in the quasi-periodic state and a failure to transition to the chaotic state. Additionally, the effects of heave and pitch amplitude variations on the dynamic characteristics of the system are also discussed.