Since January 2020, the COVID-19 pandemic has been impacting various aspects of people’s daily lives and the economy. The first case of COVID-19 in South Korea was identified on 20 January 2020. The Korean government implemented the first social distancing measures in the first week of March 2020. As a result, energy consumption in the industrial, commercial and educational sectors decreased. On the other hand, residential energy consumption increased as telecommuting work and remote online classes were encouraged. However, the impact of social distancing on residential energy consumption in Korea has not been systematically analyzed. This study attempts to analyze the impact of social distancing implemented as a result of COVID-19 on residential energy consumption with time-varying reproduction numbers of COVID-19. A two-way fixed effect model and demographic characteristics are used to account for the heterogeneity. The changes in household energy consumption by load shape group are also analyzed with the household energy consumption model. There some are key results of COVID-19 impact on household energy consumption. Based on the hourly smart meter data, an average increase of 0.3% in the hourly average energy consumption is caused by a unit increase in the time-varying reproduction number of COVID-19. For each income, mid-income groups show less impact on energy consumption compared to both low-income and high-income groups. In each family member, as the number of family members increases, the change in electricity consumption affected by social distancing tends to decrease. For area groups, large area consumers increase household energy consumption more than other area groups. Lastly, The COVID-19 impact on each load shape is influenced by their energy consumption patterns.