Rural livelihoods in semi-arid Pakistan are increasingly exposed to climate impacts such as rising temperatures, erratic rainfalls and more intense and frequent climate-related extreme events. This is introducing new risks to the already vulnerable and marginalised societies that lack development and have high poverty rates. This study uses the IPCC Livelihood Vulnerability Index (LVI) approach to analyse the determinants of household livelihood vulnerability defining vulnerability in terms of exposure, sensitivity and adaptive capacity. It also determines various adaptation responses that farmers apply and elucidates the reasons why some farmers choose not to adapt to climate change. It focuses on three semi-arid districts in Pakistan (Faisalabad, D.G. Khan and Mardan) and uses a sample of 150 rural agricultural households. As per the LVI scores, D.G. Khan is the most vulnerable district to climate change impacts, followed by Mardan and Faisalabad, respectively. Results show that (a lack of) adaptive capacity plays quite an important role in shaping households' livelihood vulnerability for any given degree of exposure and sensitivity. Besides lower exposure and sensitivity to climate change, extremely low levels of adaptive capacity make Mardan more vulnerable to climate change compared to Faisalabad. The paper argues on people-centric development for rural areas through strengthening of agriculture sector as well as providing rural household opportunities for off-farm livelihoods.