Serum neutralizing antibodies (nAbs) induced by vaccination have been linked to protection against symptomatic COVID-19 and severe disease. However, much less is known about the efficacy of nAbs in preventing the acquisition of infection, especially in the context of natural immunity and against SARS-CoV-2 immune-escape variants. In this study, we conducted mediation analysis to assess serum nAbs induced by prior SARS-CoV-2 infections as potential correlates of protection (CoPs) against Delta and Omicron BA.1/2 wave infections, in rural and urban household cohorts in South Africa. We find that, in the Delta wave, anti-D614G nAbs mediate 37% (95%CI 34% - 40%) of the total protection against infection conferred by prior exposure to SARS-CoV-2, and that protection decreases with immunity waning. In contrast, anti-Omicron BA.1 nAbs mediate 11% (95%CI 9 - 12%) of the total protection against Omicron BA.1/2 wave infections, due to Omicron's neutralization escape. These findings underscore that CoPs mediated through nAbs are variant-specific, and that boosting of nAbs against circulating variants might restore or confer immune protection lost due to nAb waning and/or immune escape. However, the majority of immune protection against SARS-CoV-2 conferred by natural infection cannot be fully explained by serum nAbs alone. Measuring these and other immune markers including T-cell responses, both in the serum and in other compartments such as the nasal mucosa, may be required to comprehensively understand and predict immune protection against SARS-CoV-2.