This study aims to promote the resource utilization of solid waste in asphalt pavements and improve its fatigue performance. For this, phosphogypsum whisker (PSW) was used as a substitute filler to prepare asphalt mixture. Through pavement performance tests and four‐point bending test, the effects of different filler types and substitution ratios on the pavement performance and fatigue performance of asphalt mixtures were investigated. On this basis, the phenomenological method was used to further construct the fatigue performance prediction model. The results indicate that PSW can effectively enhance the high temperature performance and moisture stability of the mixtures but has an adverse effect on low temperature performance. Additionally, the enhancement effect of PSW on pavement performance is more significant than PC42.5 cement (PCC). Both PSW and PCC can enhance the fatigue performance of the mixtures, with PSW exhibiting a more pronounced effect, and the incorporation of substitute fillers helps to curb the degradation trend in fatigue performance. Comparative studies indicate that the error rate of prediction model for various mixtures is significantly below 5%, with low volatility in the prediction results. Furthermore, the variance analysis results reveal that strain has a more pronounced impact on the fatigue performance of the mixtures.