Current analytical models of the mammalian immune system typically assume a specialist predator-prey relationship between invading pathogens and the active components of the immune response. However, in reality, the specific immune system is not immediately effective following invasion by a novel pathogen. First, there may be an explicit time delay between infection and immune initiation and, second, there may be a gradual build-up in immune efficacy (for instance, during the period of B-cell affinity maturation) during which the immune response develops, before reaching maximal specificity to the pathogen. Here, we use a novel theoretical approach to show that these processes, together with the presence of long-lived immune memory, decouple the immune response from current pathogen levels, greatly changing the dynamics of the pathogen-immune system interaction and the ability of the immune response to eliminate the pathogen. Furthermore, we use this model to show how distributed primary immune responses combine with immune memory to greatly affect the optimal virulence of the pathogen, potentially resulting in the evolution of highly virulent pathogens.