Polymer concrete (PC) is considered a promising repair material for asphalt pavement, since it has excellent paving performance and water stability. Although the mechanical properties of PC have been widely researched, the fatigue behavior of PC under traffic loads was still poorly understood. To predict the fatigue life and optimize the material design of PC, the semi-circular bending (SCB) tests were performed, considering different polymer content, sand ratio, aggregate features and stress condition. Two typical polymer materials were applied to prepare PC specimens, including epoxy resin (ER) and polyurethane (PU). The aggregate features were analyzed by the aggregate image measurement system. The mechanical behavior under repeated loads was investigated by the displacement, fatigue life and stiffness modulus. Results show that the flexural strength increases nonlinearly with the increasing polymer content, rapidly at first, and then slowly. The optimized polymer content and sand ratio were respectively 15% and 30%. As the loading number increases, the vertical displacement of PC shows three stages, i.e., undamaged stage, damage development stage, and fatigue failure stage. The stiffness modulus of the specimen is stress-dependent. An empirical model was developed to predict the fatigue life of PC, which can effectively capture the effects of the polymer content, sand ratio and stress level (or nominal stress ratio). It suggests that the fatigue life has a strong correlation with the mixing gradation, and the optimal sand ratio of PC can be determined by the proposed function. Moreover, the effect of aggregate shapes cannot be neglected.