We investigate the universal behavior of the Kardar-Parisi-Zhang equation with temporally correlated noise. The presence of time correlations in the microscopic noise breaks the statistical tilt symmetry, or Galilean invariance, of the original KPZ equation with delta-correlated noise (denoted SR-KPZ). Thus it is not clear whether the KPZ universality class is preserved in this case. Conflicting results exist in the literature, some advocating that it is destroyed even in the limit of infinitesimal temporal correlations, while others find that it persists up to a critical range of such correlations. Using non-perturbative and functional renormalization group techniques, we study the influence of two types of temporal correlators of the noise: a short range one with a typical time-scale τ , and a power-law one with a varying exponent θ. We show that for the short-range noise with any finite τ , the symmetries (the Galilean symmetry, and the time-reversal one in D = 1 + 1) are dynamically restored at large scales, and the long-distance properties are governed by the SR-KPZ fixed point. In the presence of a power-law noise, we find that the SR-KPZ fixed point is still stable for θ below a critical value θc, in accordance with previous RG results, while a long-range (LR) fixed-point controls the critical scaling for θ > θc, and we evaluate the θ-dependent critical exponents at this LR fixed point, in both D = 1 + 1 and D = 2 + 1 dimensions. While the results in D = 1 + 1 can be compared to previous estimates, no other prediction was available in D = 2 + 1.