The quantum speed limit (QSL) is the theoretical lower limit of the time for a quantum system to evolve from a given state to another one. Interestingly, it has been shown that non-Markovianity can be used to speed-up the dynamics and to lower the QSL time, although this behaviour is not universal. In this paper, we further carry on the investigation on the connection between QSL and non-Markovianity by looking at the effects of P- and CP-divisibility of the dynamical map to the quantum speed limit. We show that the speed-up can also be observed under P- and CP-divisible dynamics, and that the speed-up is not necessarily tied to the transition from P-divisible to non-P-divisible dynamics.