The fatigue performance of hard asphalt is an important factor that affects the service life of asphalt pavement. In order to comprehensively explore the influence of chemical components on the fatigue performance of hard asphalt, and to eliminate the chemical instability between the microstructure of asphalt from different oil sources, seven kinds of hard asphalt were designed and prepared with saturates, aromatics, resins, and asphaltenes (SARA) extracted from the same hard asphalt. Rheological, time sweep and linear amplitude sweep (LAS) tests were carried out to evaluate the fatigue properties. The results show that the complex modulus of asphalt binds increased rapidly with an increase of asphaltene and resins and that the colloidal structure was strengthened, which would increase the fatigue factor. In the time sweep test, the strength of the colloidal structure significantly affected the fatigue life, and the fatigue life was different under different test stresses. In the viscoelastic continuum damage (VECD) model, the cumulative damage was related to the modulus, while with the increase of asphaltene and resins, the fatigue life showed a trend of first increasing and then decreasing. The linear regression analysis showed that the fatigue life of hard asphalt had a good correlation with strain sensitivity. This study investigated the applicability of different fatigue evaluation methods and revealed the influence of four components on the fatigue properties of hard asphalt. The results provide significant insights in the improvement of the fatigue performance of both hard asphalt and corresponding mixtures.