Realistic representation of land carbon sink in climate models is vital for predicting carbon climate feedbacks in a changing world. Although soil erosion that removes land organic carbon has increased substantially since the onset of agriculture, it is rarely included in the current generation of climate models. Using an Earth system model (ESM) with soil erosion represented, we estimated that on average soil erosion displaces 5% of newly fixed land organic carbon downslope annually in the continental United States. In the lower Mississippi river basin and the Cascades, the fraction can be as large as 40%. About 12% of the eroded organic carbon is eventually exported to inland waters, which is equal to 14% of the simulated net carbon gain by terrestrial ecosystems. By comparing the eroded organic carbon export to rivers with the particulate organic carbon export to oceans, we demonstrated that a large fraction of the carbon export to rivers could have been mineralized in inland waters. Importantly, with a direct comparison of eroded and exported soil organic carbon and land net carbon uptake, we found that ESMs that ignore soil erosion likely offset the erosional carbon loss by increasing heterotrophic respiration implicitly. But as soil erosion and heterotrophic respiration respond differently to a warming climate, this unrealistic compensation would lead to biased predictions of future land carbon sink.