Potato (Solanum tuberosum) is an indispensable commodity, mainly cultivated by high-altitude mountain households, that sustains and supports the livelihood of an overwhelming 51% of the Bhutanese population. The popularity of potato cultivation among Bhutanese farmers can be attributed to the crop’s adaptability to a wide range of agroclimatic conditions such as a rainfed crop, high productivity, an assured market, and a reliable source of income for the farming families. We hypothesize that the changing climate would make the livelihood associated with potato cultivation in Bhutan more vulnerable. We tested this hypothesis to identify the sources of vulnerability of smallholder farming households using the Livelihood Vulnerability Index (LVI) and LVI-IPCC (Intergovernmental Panel on Climate Change) approaches in six potato growing districts of Bhutan: Bumthang, Chukha, Gasa, Mongar, Tashigang, and Wangdue. Primary data were generated through a semi structured sample survey of 240 households on the seven major livelihood components of sociodemographic profiles, livelihood strategies, social networks, health, food, water, natural disasters, and climate variability. The results showed that the LVI (range 0.302 to 0.375) and LVI-IPCC (range −0.005 to 0.030) differed significantly (p < 0.001) across the districts. The districts of Tashigang and Mongar were less vulnerable than the other four districts by the LVI approach, whereas Bumthang was also revealed to be less vulnerable using the LVI-IPCC approach. The degree of vulnerability in a district differed according to their level of exposure and adaptive capacity to the climate change impacts of the potato farming household. The results are expected to serve as empirical evidence for designing a future course of actions to mitigate the negative impacts.