Climate change significantly impacts the livelihoods of farmer households. Particularly vulnerable areas, both economically and environmentally, face significant threats from climate change. This study developed a framework to assess household-level vulnerability to climate change by integrating the Exposure-Sensitivity-Resilience Analysis (ESRA) and Sustainable Livelihoods Analysis (SLA) frameworks. Using Gui-Zhou Province as the study area, the study examined whether livelihood vulnerability differs among various types of farmer households in economically and environmentally vulnerable areas and identified the main factors contributing to vulnerability. Results indicate significant differences in livelihood vulnerability among the three household types, with pure agricultural households (PAH) being the most vulnerable due to high exposure, sensitivity, and low adaptive capacity. Further analysis revealed minor differences in sensitivity but significant differences in adaptive capacity among the three farmer categories. In terms of sensitivity, all three farmer household categories exhibit high sensitivity to water, housing, and agricultural production. Regarding adaptive capacity, significant differences in human and financial capital exist among the three household types, with off-farm households (OFH) possessing the highest adaptive capacity due to their substantial human and financial capital. Further research identified high exposure and low adaptive capacity as the primary causes of livelihood vulnerability, noting no significant difference in the main contributing factors among the three types of farmer households. Common factors contributing to the livelihood vulnerability of farmer households include agricultural cooperatives, labor capacity, temperature changes, drought frequency changes, precipitation changes, agricultural insurance, and losses in agricultural production. Overall, the proposed livelihood vulnerability framework offers guidance for analyzing the vulnerability of farmer households in areas with both economic and environmental vulnerabilities under climate change. Concurrently, tailored measures to reduce farmer households’ livelihood vulnerability should be developed for different household types, considering the local climatic, geographic, and socioeconomic conditions.