Random genetic drift in the population-level dynamics of an infectious disease outbreak results from the randomness of inter-host transmission and the randomness of host recovery or death. The strength of genetic drift has been found to be high for SARS-CoV-2 due to superspreading, and this is expected to substantially impact the disease epidemiology and evolution. Noise that results from the measurement process, such as biases in data collection across time, geographical areas, etc., can potentially confound estimates of genetic drift as both processes contribute "noise" to the data. To address this challenge, we develop and validate a method to jointly infer genetic drift and measurement noise from time-series lineage frequency data. We apply this method to over 490,000 SARS-CoV-2 genomic sequences from England collected between March 2020 and December 2021 by the COVID-19 Genomics UK (COG-UK) consortium. We find that even after correcting for measurement noise, the strength of genetic drift is consistently, throughout time, higher than that expected from the observed number of COVID-19 positive individuals in England by 1 to 3 orders of magnitude. Corrections taking into account epidemiological dynamics (susceptible-infected-recovered or susceptible-exposed-infected-recovered models) do not explain the discrepancy. Moreover, the levels of genetic drift that we observe are higher than the estimated levels of superspreading found by modeling studies that incorporate data on actual contact statistics in England. We discuss how even in the absence of superspreading, high levels of genetic drift can be generated via community structure in the host contact network. Our results suggest that further investigations of heterogeneous host contact structure may be important for understanding the high levels of genetic drift observed for SARS-CoV-2 in England.