Variations in viral loads generated during SARS-CoV-2 infections can influence COVID-19 disease progression and the likelihood of onward transmission. However, the host and virus factors that may impact viral loads and infection dynamics are not fully understood. Here, we conducted virus whole genome sequencing and measured viral copies using RT-qPCR from 9,902 SARS-CoV-2 infections over a 2-year period to examine the relative impact of host factors and virus genetic variation on changes in viral copies. We first used statistical regression models to show that host age and SARS-CoV-2 variant significantly impact viral copies, but vaccination status does not. Then, using a genome-wide association study (GWAS) approach, we identified multiple nucleotide substitutions corresponding to amino acid changes in the SARS-CoV-2 genome associated with variations in viral copies. In particular, we analyzed the temporal patterns and found that SNPs associated with higher viral copies were predominantly observed in Omicron BA.2/BA.4/BA.5/XBB infections, whereas those associated with decreased viral copies were mostly observed in infections with Delta and Omicron BA.1 variants. Our work showcases how GWAS can be a useful tool for probing phenotypes related to SNPs in viral genomes, which can be used to characterize emerging variants and monitor therapeutic interventions.