A polyurethane mixture (PUM) is an energy-saving and emission-reducing pavement material with excellent temperature stability; however, the fatigue properties and fatigue damage models of PUM still require further research. Therefore, four-point bending static load tests, fatigue tests, and digital speckle correlation method (DSCM) tests with different load levels of PUM and styrene butadiene styrene (SBS)-modified asphalt mixture (SMA) were carried out. The fatigue life, stiffness, midspan deflection, and maximum tensile strain were obtained and compared. The fatigue damage factor calculation method of PUM based on stiffness degradation was proposed, and the fatigue damage function of PUM at different load levels was fitted. The results show that the fatigue life of PUM was much larger than that of SMA, and the static loading failure and fatigue failure modes of PUM were both brittle. The fatigue damage of PUM exhibits an obvious three-stage damage law: the rapid development stage (accounting for about 10–20% of the fatigue life), the deformation stability expansion stage (accounting for about 70–80% of the fatigue life), and the instability development stage (accounting for about 10–20% of the fatigue life). The fatigue damage factors (DB) were calculated based on stiffness, according to DB=EI0−EInrEI0−EINr, and the fatigue damage functions of PUM were fitted based on the stiffness degradation, according to fnN=1−1−(n/N)a(1−n/N)b. The fatigue damage fitting curves have good correlation with the calculation results of the damage factor based on test data, which can predict the stiffness degradation of PUM at different load levels. The results can help further the understanding of the fatigue characteristics and damage mechanism of PUM, which will provide theoretical support for the application of PUM in pavement structures.