Since the emergence of SARS-CoV-2 Omicron BA.1 and BA.2, several Omicron sublineages have emerged, supplanting their predecessors. BA.5 is the current dominant sublineage. Here we compared the neutralization of Omicron sublineages BA.1, BA.2, BA.4 and BA.5 by human sera collected from individuals who were infected with the ancestral B.1 (D614G) strain, vaccinated (3 doses), or with hybrid immunity from vaccination (2 doses) followed by pre-Omicron breakthrough infection (BTI) with Gamma or Delta. All Omicron sublineages exhibited extensive escape from all sera compared to the ancestral B.1 strain and to Delta, albeit to different levels depending on the origin of the sera. Convalescent sera were unable to neutralize BA.1, and partly neutralized BA.2, BA.4 and BA.5. Vaccinee sera partly neutralized BA.2, but BA.1, BA.4 and BA.5 evaded neutralizing antibodies. BTI sera were either non-neutralizing or partially neutralizing. In this case, they had similar neutralizing ability against all Omicron sublineages. Despite similar levels of anti-Spike and anti-Receptor Binding Domain (RBD) antibody in all groups, BTI sera had the highest cross-neutralizing ability against all Omicron sublineages and convalescent sera were the least neutralizing. The NT50:antibody titer ratio, which reflects antibody avidity, was significantly higher in sera from BTI patients compared to convalescent sera, underscoring qualitative differences in antibodies elicited by infection alone and by vaccination. Together these findings highlight the importance of vaccination to trigger highly cross-reactive antibodies that neutralize phylogenetically and antigenically distant strains, and suggest that immune imprinting by first generation vaccines may restrict, but not abolish cross-neutralization.