The evaluation of the neutralizing capacity of anti-SARS-CoV-2 antibodies is important because they represent real protective immunity. In this study we aimed to measure and compare the neutralizing antibodies (NAbs) in COVID-19 patients and in vaccinated individuals. One-hundred and fifty long-term samples from 75 COVID-19 patients were analyzed with a surrogate virus neutralization test (sVNT) and compared to six different SARS-CoV-2 serology assays. The agreement between the sVNT and pseudovirus VNT (pVNT) results was found to be excellent (i.e., 97.2%). The NAb response was also assessed in 90 individuals who had received the complete dose regimen of BNT162b2. In COVID-19 patients, a stronger response was observed in moderate–severe versus mild patients (p-value = 0.0006). A slow decay in NAbs was noted in samples for up to 300 days after diagnosis, especially in moderate–severe patients (r = −0.35, p-value = 0.03). In the vaccinated population, 83.3% of COVID-19-naive individuals had positive NAbs 14 days after the first dose and all were positive 7 days after the second dose, i.e., at day 28. In previously infected individuals, all were already positive for NAbs at day 14. At each time point, a stronger response was observed for previously infected individuals (p-value < 0.05). The NAb response remained stable for up to 56 days in all participants. Vaccinated participants had significantly higher NAb titers compared to COVID patients. In previously infected vaccine recipients, one dose might be sufficient to generate sufficient neutralizing antibodies.