Abrasive blast cleaning is often done for steel structures before applying various protective coatings, which produces rough surfaces with changes in fatigue properties. This problem has been addressed in the low- and high-cycle fatigue regimes; however, the effect of surface roughness in combination with different loading parameters on the ultra-low cycle fatigue (ULCF) life has not been reported thus far. To this aim, a total of 59 ULCF tests on designed specimens of SM400 steel with five levels of surface roughness were performed under various loading frequencies and displacement amplitudes. The analysis of experimental results indicates a substantial reduction in fatigue life with an increase in the surface roughness and loading amplitude and a decrease in the loading frequency. Additionally, the strength degradation, dissipation energy, and load–displacement curves are discussed in detail. With the use of experimental data, a new life prediction model characterizing the combined effects of surface roughness, loading frequency, and loading amplitude on the ULCF life is proposed. Moreover, the proposed model is validated by predicting the fatigue life under variable and constant loading amplitude patterns. Comparison between experimental and theoretical results shows that the proposed model accurately estimates the ULCF life within an error band of ±15%, with a reasonable selection of model parameters.