Based on the survey data on animal husbandry from 1,689 households in semi-agricultural and semi-pastoral counties in Inner Mongolia, this paper applied the “buffer capacity–organizational capacity-learning capacity” framework to analyze the current livelihood resilience of farmers and herdsmen, as well as the impact of ecological vulnerability and disaster shocks on this resilience. The results show that, first, due to the vicious ecological environment and natural disasters, livelihood resilience among farmers and herdsmen is generally low in the region, but that of herdsmen is significantly higher than that of farmers. There are clear differences between the dimensions of livelihood resilience in different households. Second, natural disasters, of which drought is the most obvious, have a great impact on livelihood resilience. However, there is a significant positive correlation between ecological vulnerability and the livelihood resilience of farmers and herdsmen; thus, we should reflect on the past development model of the region. Third, In addition to the impact of ecological vulnerability and disaster shocks, per capita income, human capital, policy support, social networks, and information access are the main obstacles to livelihood resilience. Combined with these research findings, this paper seeks to improve livelihood resilience through the strategies of avoiding disaster risk, changing the development mode, reducing path dependence, and identifying obstacles.