Although the fatigue properties of asphalt materials have been extensively studied, the relationship between the rheological properties and road performance of asphalt mixtures remains underexplored. In this study, we have examined the relaxation properties of asphalt binders through relaxation tests conducted on asphalt and its mastic under different conditions. A repeated stress relaxation-recovery test is designed for assessing both the relaxation and elastic properties, and a set of reasonable test parameters is recommended, thereby establishing a novel test method for measuring the relaxation and elastic behaviors of asphalt. In addition, we have proposed evaluation indexes, such as relaxation time, relaxation rate, and strain recovery rate, to assess the stress relaxation performance and strain recovery ability. It is observed that the relaxation rate and strain recovery rate of the material can be used to characterize the material’s relaxation and elasticity properties, respectively. Thus, the proposed indexes can be used to comprehensively evaluate the viscoelastic performance of the material. The fatigue performance of the selected materials is further examined using the linear amplitude sweep (LAS) test, and the correlation between the relaxation properties and fatigue performance (as indicated by fatigue parameters) is explored. Furthermore, a fatigue performance prediction equation based on the repetitive stress relaxation-recovery test is established. The findings reveal a strong correlation between the relaxation properties and fatigue performance, suggesting that the stress relaxation test can accurately assess the fatigue performance of asphalt materials.