Domestic herbivores have been closely associated with the historical evolution and development of agriculture systems worldwide as a complementary system for providing milk, meat, wool, leather, and animal power. However, their major role was to enhance and maintain agricultural soil fertility through the recycling of nutrients. In turn, cereal production increased, enabling to feed a progressively increasing human population living in expanding urban areas. Further, digestion of organic matter through the rumen microbiome can also be viewed as enhancing the soil microbiome activity. In particular, when animal droppings are deposited directly in grazing areas or applied to fields as manure, the mineralization–immobilization turnover determines the availability of nitrogen, phosphorus, potassium, and other nutrients in the plant rhizosphere. Recently, this close coupling between livestock production and cereal cropping systems has been disrupted as a consequence of the tremendous use of industrial mineral fertilizers. The intensification of production within these separate and disconnected systems has resulted in huge emissions of nitrogen (N) to the environment and a dramatic deterioration in the quality of soil, air, and ground- and surface water. Consequently, to reduce drastically the dependency of modern and intensified agriculture on the massive use of N and phosphorus (P) fertilizers, we argue that a close reconnection at the local scale, of herbivore livestock production systems with cereal-based cropping systems, would help farmers to maintain and recover the fertility of their soils. This would result in more diverse agricultural landscapes including, besides cereals, grasslands as well as forage and grain crops with a higher proportion of legume species. We developed two examples showing such a beneficial reconnection through (i) an agro-ecological scenario with profound agricultural structural changes on a European scale, and (ii) typical Brazilian integrated crop–livestock systems (ICLS). On the whole, despite domestic herbivores emit methane (CH4), an important greenhouse gas, they participate to nutrient recycling, which can be viewed as a solution to maintaining long-term soil fertility in agro-ecosystems; at a moderate stocking density, ecosystem services provided by ruminants would be greater than the adverse effect of greenhouse gas (GHG).