Surface roughness and slope gradient are two important factors influencing soil erosion. The objective of this study was to investigate the interaction of surface roughness and slope gradient in controlling soil loss from sloping farmland due to water erosion on the Loess Plateau, China. Following the surface features of sloping farmland in the plateau region, we manually prepared rough surfaces using four tillage practices (contour drilling, artificial digging, manual hoeing, and contour plowing), with a smooth surface as the control measure. Five slope gradients (3°, 5°, 10°, 15°, and 20°) and two rainfall intensities (60 and 90 mm/hr) were considered in the artificial rainfall simulation experiment. The results showed that the runoff volume and sediment yield increased with increasing slope gradient under the same tillage treatment. At gentle slope gradients (e.g., 3° and 5°), the increase in surface roughness prevented the runoff and sediment production, that is, the surface roughness reduced the positive effect of slope gradient on the runoff volume and sediment yield to a certain extent. At steep slope gradients, however, the enhancing effect of slope gradient on soil erosion gradually increased and surpassed the reduction effect of surface roughness. This study reveals the existence of a critical slope gradient that influences the interaction of surface roughness and slope gradient in controlling soil erosion on sloping farmland. If the slope gradient is equal to or less than the critical value, an increase in surface roughness would decrease soil erosion. Otherwise, the increase in surface roughness would be ineffective for preventing soil erosion. The critical slope gradient would be smaller under higher rainfall intensity. These findings are helpful for us to understand the process of soil erosion and relevant for supporting soil and water conservation in the Loess Plateau region of China.