It is important to couple the analysis of the effects of topography, climate, vegetation, and human activities on soil erosion to understand the dynamic characteristics of soil erosion under environmental change. This study investigated the impacts of geographic unit difference on soil erosion in the Shaanxi Province in China through the model simulation (RUSLE) and quantitative analysis. The following results were achieved. First, the amplitudes of environmental change varied at different rates in the three regions over the past 35 years (1980–2015). The frequency of severe rainstorms, the main trigger for soil erosion, intensified most in the Loess Plateau compared to the Qinling Mountains and Guanzhong Plain and led to an increase in the number of geological disasters in that region. In addition, after the 1990s, vegetation improved most in the Loess Plateau, and socioeconomic activities related to urbanization, another important driver of soil erosion, increased the most in Guanzhong Plain. Second, improved vegetation cover was the most significant contributor to the reduction in soil erosion during the past 35 years, although intensified rainstorms and human activities enhanced the risks of soil erosion. Third, the comparative study revealed that different environmental parameters dominated soil erosion processes in the three regions.