SARS-CoV-2 variants raise concern because of their high transmissibility and their ability to evade neutralizing antibodies elicited by prior infection or by vaccination. Here we compared the ability of sera from 70 unvaccinated COVID-19 patients infected before the emergence of variants of concern (VOCs) and from 16 vaccine breakthrough infection (BTI) cases infected with Gamma or Delta to neutralize the ancestral B.1 strain, and the Gamma, Delta and Omicron BA.1 variants using live virus. We further determined antibody levels against the Spike protein, the Receptor Binding Domain (RBD) and the N-terminal domain (NTD) of Spike. Convalescent sera featured considerable variability in neutralization of B.1 and in cross-neutralization of different strains, and neutralizing capacity moderately correlated with antibody levels against Spike and the RBD. All but one convalescent serum failed to neutralize Omicron BA.1. Overall, convalescent sera from patients with moderate disease had higher antibody levels and displayed higher neutralizing ability against all strains than patients with mild or severe forms of disease. Sera from BTI cases fell into one of two categories: half the sera had high neutralizing activity against the ancestral B.1 strain as well as against the infecting strain, while the other half had no or very low neutralizing activity against all strains. Although antibody levels against Spike and the RBD were lower in BTI cases than in unvaccinated convalescent sera, most neutralizing sera also retained partial neutralizing activity against Omicron BA.1, indicative of cross-neutralization between B.1, Delta and Omicron and suggestive of higher affinity, as confirmed by the IC50:Ab level ratios. Neutralizing activity of BTI sera was strongly correlated with antibodies against Spike and the RBD. Together, these findings highlight qualitative differences in antibody responses elicited by infection in vaccinated and unvaccinated individuals. They further suggest that breakthrough infection with a pre-Omicron variant boosts immunity and induces cross neutralizing antibodies against different strains, including Omicron BA.1.