The effectiveness of antiviral treatment with remdesivir against COVID-19 has been investigated in clinical trials suggesting earlier recovery. However, this effect seems to be rather modest. In this study, we assessed the clinical course of SARS-CoV-2 infections in 369 COVID-19 individuals across a spectrum of illness severities, including both untreated individuals and individuals who received antiviral treatment with remdesivir. Moreover, using a process-based mathematical model, we quantified and analyzed the within-host infection dynamics of 69 untreated and 19 antiviral-treated individuals. For untreated individuals, we found that those hospitalized exhibit significantly lower levels of early immune response and higher cumulative viral loads than those who were not. For treated individuals, we found that those who died were on average hospitalized later after symptom onset than those who survived, underscoring the importance of early medical intervention for severe COVID-19. Our model estimates a rather limited antiviral activity of remdesivir and, consequently, comparable viral load dynamics between individuals responding and not responding to antiviral treatment. Our results provide valuable insights into the clinical course of COVID-19 during antiviral treatment with remdesivir and suggest the need for alternative treatment regimens.