Background: Prior infection with SARS-CoV-2 can provide protection against infection and severe COVID-19. In settings with high pre-existing immunity, vaccine effectiveness (VE) should decrease with higher levels of immunity among unvaccinated individuals. Here, we conducted a systematic review and meta-analysis to understand the influence of prior infection on VE. Methods: We included test-negative design (TND) studies that examined VE against infection or severe disease (hospitalization, ICU admission, or death) for primary vaccination series. To determine the impact of prior infections on VE estimates, we compared studies that excluded or included people with prior COVID-19 infection. We also compared VE estimates by the cumulative incidence of cases before the start of and incidence rates during each study in the study locations, as further measures of prior infections in the community. Findings: We identified 67 studies that met inclusion criteria. Pooled VE among studies that included people with prior COVID-19 infection was lower against infection (pooled VE: 77%; 95% confidence interval (CI): 72%, 81%) and severe disease (pooled VE: 86%; 95% CI: 83%, 89%), compared with studies that excluded people with prior COVID-19 infection (pooled VE against infection: 87%; 95% CI: 85%, 89%; pooled VE against severe disease: 93%; 95% CI: 91%, 95%). There was a negative correlation between the cumulative incidence of cases before the start of the study and VE estimates against infection (spearman correlation (ρ) = -0.32; 95% CI: -0.45, -0.18) and severe disease (ρ = -0.49; 95% CI: -0.64, -0.30). There was also a negative correlation between the incidence rates of cases during the study period and VE estimates against infection (ρ = -0.48; 95% CI: -0.59, -0.34) and severe disease (ρ = -0.42; 95% CI: -0.58, -0.23). Interpretation: Based on a review of published VE estimates we found clear empirical evidence that higher levels of pre-existing immunity in a population were associated with lower VE estimates. Excluding previously infected individuals from VE studies may result in higher VE estimates with limited generalisability to the wider population. Prior infections should be treated as confounder and effect modificatory when the policies were targeted to whole population or stratified by infection history, respectively.