During the first months of HIV infection, the virus typically evolves several immune escape mutations. These mutations are found in epitopes in viral proteins and reduce the impact of the CD8+ T cells specific for these epitopes. Recent data show that only a subset of the epitopes escapes, that most of these escapes evolve early, and that the rate of immune escape slows down considerably. To investigate why the evolution of immune escape slows down over the time of infection, we have extended a consensus mathematical model to allow several immune responses to control the virus together. In the extended model, most escapes also occur early, and the immune escape rate becomes small later, and typically only a minority of the epitopes escape. We show that escaping one of the many immune responses provides little advantage after viral setpoint has been approached because the total killing rate hardly depends on the breadth of the immune response. If the breadth of the immune response slowly wanes during disease progression, the model predicts an increase in the rate of immune escape at late stages of infection. Overall, the most striking prediction of the model is that HIV evolves a small number of immune escapes, in both relative and absolute terms, when the CTL immune response is broad.