The crankshaft is one of the core components of a Rotate Vector (RV) reducer. The fatigue life of the RV reducer is severely hindered by fatigue failure on the eccentric cylindrical surface of the crankshaft. The hardness gradients and residual stress in the crankshaft, associated with machining operations, exert an enormous impact on the rolling contact fatigue (RCF). In this work, a finite element method (FEM)-based three-dimensional elasto-plastic contact model is established to calculate the stress–strain field by taking hardness gradients and initial residual stress into account. The RCF characteristics of an RV reducer crankshaft is investigated by applying modified Fatemi–Socie (FS) multiaxial fatigue criterion. The results indicate that initial residual stress plays an influential role in the fatigue damage by altering the distribution of the maximum normal stress near the contact surface. The modified FS fatigue criterion could better consider the effect of initial residual stress and the shear stress, which significantly improves the prediction accuracy of the contact fatigue life model. The contact fatigue performance could be considerably improved by designing appropriate shot peening parameters to obtain optimized residual stress distribution. Therefore, the technique presented may serve as an important guideline for the anti-fatigue design of an RV reducer crankshaft.