The COVID-19 pandemic has had unprecedented effects on our daily lives. This study aimed to assess the quality of life (QoL) (WHOQOL-Bref physical, social, and environmental domains) at two time points during the COVID-19 pandemic with lockdown restrictions according to gender, age, and urbanization level. Qualtrics® recruited representative Austrian population samples in April 2020 (t1; N = 1,005) and December 2020/January 2021 (t2; N = 1,505). ANOVAs and the Bonferroni-corrected post-hoc tests were conducted to investigate differences between April and December 2020 and to compare with pre-pandemic data. Although the quality of life (physical, social, and environmental domains) changed from pre-pandemic (mean scores 80, 77, and 81, respectively) to April 2020 (mean scores 72, 65, and 75, all p-values < 0.001), there were no significant changes between April and December (mean scores 75, 65, and 75). Living location (urban vs. rural), gender, and age showed an effect on the quality of life. All domains of quality of life have decreased since the onset of the pandemic, and this decline has been maintained over the course of the first year of the pandemic. Creative measures should be implemented to assist people in improving one or more areas of quality of life, within the lockdown restrictions to improve the overall wellbeing of the population.