Fatigue crack in a rotary shaft is a common failure observed in rotor systems. Since vibration of the shaft causes alternating fatigue loads, the crack propagates slowly. Meanwhile, the propagating crack may cause nonlinear or unstable vibration of the rotor system. In fact, growth of the crack and vibration of the shaft are coupled with each other. Hence, it is necessary to study the fatigue degradation behavior of the cracked rotor accounting for this coupling effect. In this paper, a coupling model of rotor vibration and crack growth is established through dynamic theory and fracture mechanics, and a sequential iterative procedure is proposed to solve the coupling model. Then, the competing degradation failure mode of the cracked rotor is analyzed with regard to the rapid crack growth failure and the unstable vibration failure. And degradation measures are proposed based on the competing degradation failure criterion. At last, degradation behaviors with the coupling effect of nonlinear vibration behavior and multiple parameters including rotation speed, unbalance eccentricity and orientation angle, and damping are investigated by numerical simulation. The results indicate that nonlinear vibration behavior and multiple parameters have considerable influence on the degradation behaviors, which present complex regularity. The findings are of significance to guide the safety design of the rotor system for long time operation and help to the further research on prognostics and lifetime prediction.