Vaccination against COVID-19 was integral to controlling the pandemic that persisted with the continuous emergence of SARS-CoV-2 variants. Using a mathematical model describing SARS-CoV-2 within-host infection dynamics, we estimate differences in virus and immunity due to factors of infecting variant, age, and vaccination history (vaccination brand, number of doses and time since vaccination). We fit our model in a Bayesian framework to upper respiratory tract viral load measurements obtained from cases of Delta and Omicron infections in Singapore, of whom the majority only had one nasopharyngeal swab measurement. With this dataset, we are able to recreate similar trends in URT virus dynamics observed in past within-host modelling studies fitted to longitudinal patient data.We found that Omicron had higher R0,within values than Delta, indicating greater initial cell-to-cell spread of infection within the host. Moreover, heterogeneities in infection dynamics across patient subgroups could be recreated by fitting immunity-related parameters as vaccination history-specific, with or without age modification. Our model results are consistent with the notion of immunosenescence in SARS-CoV-2 infection in elderly individuals, and the issue of waning immunity with increased time since last vaccination. Lastly, vaccination was not found to subdue virus dynamics in Omicron infections as well as it had for Delta infections.This study provides insight into the influence of vaccine-elicited immunity on SARS-CoV-2 within-host dynamics, and the interplay between age and vaccination history. Furthermore, it demonstrates the need to disentangle host factors and changes in pathogen to discern factors influencing virus dynamics. Finally, this work demonstrates a way forward in the study of within-host virus dynamics, by use of viral load datasets including a large number of patients without repeated measurements.