Ecocultural tourism plays an crucial role in promoting poverty reduction and farmers’ sustainable livelihood enhancement worldwide, which has attracted strong attention from scholars and society. However, the impact of ecological and cultural capital of farmers’ sustainable livelihoods have not been yet fully investigated. This paper modifies the sustainable livelihoods framework (SLF) and emphasizes the effects of ecological and cultural capital on farmers’ sustainable livelihoods. The modified SLF can be used as a possible theoretical model to comprehensively characterize farmers’ sustainable livelihoods in tourism destinations with rich ecological and cultural resources. By constructing an evaluation index system, principle component analysis and multiple linear regression are used to analyze the types and response levels of farmers’ sustainable livelihoods to ecocultural tourism and determine the factors influencing this response. The results show that farmers’ sustainable livelihood responses to ecocultural tourism can be classified into six types. The overall response values are low, and the responses are ranked by the degree of response in descending order as complete response, balanced development, cultural network, ecology-dominant, developmental delay, and resource advantage response. Farmers’ sustainable livelihoods respond strongly to cultural capital and ecological capital factors and relatively weakly to physical, natural, social, financial, and labor capital. Five main factors that influence farmers’ sustainable livelihood responses are then identified. Finally, strategies and suggestions for livelihood transformation, strategy selection, and improvement are proposed. This study provides case study examples for promoting the sustainable development of ecocultural tourism sites and improving farmers’ livelihoods in China and worldwide.