Climate change vulnerability is disproportionately distributed between different population segments in society. This study qualitatively explores how key stakeholders in municipalities (i.e. planning and operational staff in municipalities and the vulnerable themselves) construct social vulnerability in relation to climate change with a specific focus on thermal stress (i.e. heat waves) and which adaptive responses they identify at different levels. The empirical material consists of five focus groups with actors in a large Swedish municipality where the "Vulnerability Factor Card Game" was used as stimulus material to create 10 fictional individuals. The results show that there is a substantial amount of local knowledge about vulnerability drivers and inter-relations between social factors and vulnerability. Local decision-makers also defined a wide range of possible adaptation measures at different municipal levels. Our study clearly indicates that contextualised knowledge, which could complement the quantitative approaches in research, is abundant among municipal planners, staff employed at municipal operations such as health care, and among the vulnerable themselves. This knowledge remains untapped by research to a great extent and only seems to have an insignificant influence on policy-making. In particular, how impacts vary between different social and demographic groups and how adaptation strategies that target the most vulnerable could be defined are of great interest. The present study clearly indicates that social hierarchy may produce increased inequality in the specific context of climate change, vulnerability and adaptive responses at different levels.