It has been very difficult to predict the development of the COVID-19 pandemic based on mathematical models for the spread of infectious diseases, and due to major non-pharmacological interventions (NPIs), it is still unclear to what extent the models would have fit reality in a “do nothing” scenario. To shed light on this question, the case of Sweden during the time frame from autumn 2020 to spring 2021 is particularly interesting, since the NPIs were relatively minor and only marginally updated. We found that state of the art models are significantly overestimating the spread, unless we assume that social interactions significantly decrease continuously throughout the time frame, in a way that does not correlate well with Google-mobility data nor updates to the NPIs or public holidays. This leads to the question of whether modern SEIR-type mathematical models are unsuitable for modeling the spread of SARS-CoV-2 in the human population, or whether some particular feature of SARS-CoV-2 dampened the spread. We show that, by assuming a certain level of pre-immunity to SARS-CoV-2, we obtain an almost perfect data-fit, and discuss what factors could cause pre-immunity in the mathematical models. In this scenario, a form of herd-immunity under the given restrictions was reached twice (first against the Wuhan-strain and then against the alpha-strain), and the ultimate decline in cases was due to depletion of susceptibles rather than the vaccination campaign.