It is generally acknowledged that soil erosion has become one of the greatest global threats to the human-environment system. Although the Revised Universal Soil Loss Equation (RUSLE) has been widely used for soil erosion estimation, the algorithm for calculating soil erodibility factor (K) in this equation remains limited, particularly in the context of China, which features highly diverse soil types. In order to address the problem, a modified algorithm describing the piecewise function of gravel content and relative soil erosion was used for the first time to modify the soil erodibility factor, because it has been proven that gravel content has an important effect on soil erosion. The Chaohu Lake Basin (CLB) in East China was used as an example to assess whether our proposal can improve the accuracy of soil erodibility calculation and soil erosion estimation compared with measured data. Results show that (1) taking gravel content into account helps to improve the calculation of soil erodibility and soil erosion estimation due to its protection to topsoil; (2) the overall soil erosion in the CLB was low (1.78 Mg·ha −1 ·year −1 ) the majority of which was slight erosion (accounting for 85.6%) and no extremely severe erosion; and (3) inappropriate land use such as steep slope reclamation and excessive vegetation destruction are the main reasons for soil erosion of the CLB. Our study will contribute to decision-makers to develop soil and water conservation policies.