The fatigue resistance of asphalt mixture is an important indicator to evaluate the durability of asphalt pavement. In order to improve the fatigue properties of asphalt mixture, diatomite and environmental basalt fiber were added. Four types of asphalt mixtures, ordinary asphalt mixture (AM), diatomite modified asphalt mixture (DAM), basalt fiber modified asphalt mixture (BFAM) and diatomite/basalt fiber composite modified asphalt mixture (DBFAM), were chosen, whose optimum asphalt–aggregate ratio, optimum content of diatomite and optimum content of basalt fiber could be determined by Marshall test and response surface methodology (RSM). The multi-functional pneumatic servo Cooper test machine was carried out by a four-point bending fatigue test. Through the comparative analysis of flexural-tensile stiffness modulus (S), initial stiffness modulus(S0), residual stiffness modulus ratio, lag angle (ϕ) and cumulative dissipation energy (ECD), the fatigue resistance of asphalt mixture can be effectively improved by adding diatomite and basalt fiber. Grey correlation analysis was also used to analyze the degree of correlation between the fatigue life and the influencing factors such as VV, VMA, VFA, OAC, S, and ECD. The analysis results indicate that ECD has the greatest impact on the fatigue life of the asphalt mixture.