The apparent lack of antigenic evolution by the Delta variant (B.1.617.2) of SARS-CoV-2 during the COVID-19 pandemic is puzzling. The combination of increasing immune pressure due to the rollout of vaccines and a relatively high number of infections following the relaxation of non-pharmaceutical interventions should have created perfect conditions for immune escape variants to evolve from the Delta lineage. Instead, the Omicron variant (B.1.1.529), which is hypothesised to have evolved in an immunocompromised individual, is the first major variant to exhibit significant immune escape following vaccination programmes and is set to become globally dominant in 2022. Here, we use a simple mathematical model to explore possible reasons why the Delta lineage did not exhibit antigenic evolution and to understand how and when immunocompromised individuals affect the emergence of immune escape variants. We show that when the pathogen does not have to cross a fitness valley for immune escape to occur, immunocompromised individuals have no qualitative effect on antigenic evolution (although they may accelerate immune escape if within-host evolutionary dynamics are faster in immunocompromised individuals). But if a fitness valley exists between immune escape variants at the between-host level, then persistent infections of immunocompromised individuals allow mutations to accumulate, therefore facilitating rather than simply speeding up antigenic evolution. Our results suggest that better global health equality, including improving access to vaccines and treatments for individuals who are immunocompromised (especially in lower- and middle-income countries), may be crucial to preventing the emergence of future immune escape variants of SARS-CoV-2.