The building sector is facing a challenge in achieving carbon neutrality due to climate change and urbanization. Urban building energy modeling (UBEM) is an effective method to understand the energy use of building stocks at an urban scale and evaluate retrofit scenarios against future weather variations, supporting the implementation of carbon emission reduction policies. Currently, most studies focus on the energy performance of archetype buildings under climate change, which is hard to obtain refined results for individual buildings when scaling up to an urban area. Therefore, this study integrates future weather data with an UBEM approach to assess the impacts of climate change on the energy performance of urban areas, by taking two urban neighborhoods comprising 483 buildings in Geneva, Switzerland as case studies. In this regard, GIS datasets and Swiss building norms were collected to develop an archetype library. The building heating energy consumption was calculated by the UBEM tool—AutoBPS, which was then calibrated against annual metered data. A rapid UBEM calibration method was applied to achieve a percentage error of 2.7%. The calibrated models were then used to assess the impacts of climate change using four future weather datasets out of Shared Socioeconomic Pathways (SSP1-2.6, SSP2-4.5, SSP3-7.0, and SSP5-8.5). The results showed a decrease of 22%–31% and 21%–29% for heating energy consumption, an increase of 113%–173% and 95%–144% for cooling energy consumption in the two neighborhoods by 2050. The average annual heating intensity dropped from 81 kWh/m
2
in the current typical climate to 57 kWh/m
2
in the SSP5-8.5, while the cooling intensity rose from 12 kWh/m
2
to 32 kWh/m
2
. The overall envelope system upgrade reduced the average heating and cooling energy consumption by 41.7% and 18.6%, respectively, in the SSP scenarios. The spatial and temporal distribution of energy consumption change can provide valuable information for future urban energy planning against climate change.