Conformal contact is a commonly presented contact form in assemblies. Nonproportional loading is the main characteristics of conformal contact, which leads to prominent difficulty in revealing fretting crack behavior. In this paper, a finite element prediction model for Ti-6Al-4V pin-hole contact fretting fatigue crack initiation was developed, which simultaneously considered the effect of fretting wear, surface roughness, surface skewness, surface kurtosis, and residual stress. The results show that phase differences of stress component, change in direction of principal stress, and high stress gradient are the main reasons for the initiation of fretting fatigue under conformal contact condition. The model based on the Fatemi-Socie (FS) parameter successfully predicted the location, orientation, and fatigue life of crack initiation, which agrees well with the experimental results. Additionally, machining-induced residual stress can effectively inhibit mode I crack initiation at valleys. Moreover, ignoring the surface microgeometry characteristics reduces the prediction accuracy of the crack behavior.