Non-Markovian effect is found to be able to decrease the quantum speed limit (QSL) time, and hence to enhance the intrinsic speed of quantum evolution. Although a reservoir with larger degree of non-Markovianity may seem like it should cause smaller QSL times, this seemingly intuitive thinking may not always be true. We illustrate this by investigating the QSL time of a qubit that is coupled to a two-band photonic-band-gap (PBG) environment. We show how the QSL time is influenced by the coherent property of the reservoir and the band-gap width. In particular, we find that the decrease of the QSL time is not attributed to the increasing non-Markovianity, while the memory time of the environment can be seen as an essential reflection to the QSL time. So, the QSL time provides a further insight and sharper identification of memory time in a PBG environment. We also discuss a feasible experimental realization of our prediction.