We aimed to examine the prevalence of, and factors associated with, quality of life (QOL) worsening among coronavirus disease 2019 (COVID-19) survivors. This population-based retrospective cohort study used data from the Korea Disease Control and Prevention Agency and the National Health Insurance Service in South Korea. A total of 325,666 COVID-19 survivors were included in this study. Among them, 106,091 (32.6%) survivors experienced worsening QOL after COVID-19. Specifically, 21,223 (6.5%) participants experienced job loss, 94,556 (29.0%) experienced decreased household income, and 559 (0.2%) acquired new disabilities. In the multivariable logistic regression model, living in rural areas (odds ratio [OR]: 1.02; 95% confidence interval [CI]: 1.01, 1.04; p = 0.009), intensive care unit admission (OR: 1.08, 95% CI: 1.02, 1.15; p = 0.028), and increase in self-payment by 100 USD (OR: 1.02, 95% CI: 1.02, 1.02; p < 0.001) were associated with increased QOL worsening after COVID-19. Old age (OR: 0.99, 95% CI: 0.98, 0.99; p < 0.001), first vaccination (OR: 0.89, 95% CI: 0.86, 0.93; p < 0.001), and second vaccination (OR: 0.95, 95% CI: 0.93, 0.96; p < 0.001) were associated with decreased QOL worsening after COVID-19. Approximately one-third of COVID-19 survivors in South Korea who were admitted to hospitals or monitoring centers experienced QOL worsening.