Quantum speed limits set an upper bound to the rate at which a quantum system can evolve and as such can be used to analyze the scrambling of information. To this end, we consider the survival probability of a thermofield double state under unitary time-evolution which is related to the analytic continuation of the partition function. We provide an exponential lower bound to the survival probability with a rate governed by the inverse of the energy fluctuations of the initial state. Further, we elucidate universal features of the non-exponential behavior at short and long times of evolution that follow from the analytic properties of the survival probability and its Fourier transform, both for systems with a continuous and for systems with a discrete energy spectrum. We find the spectral form factor in a number of illustrative models, notably we obtain the exact answer in the Gaussian unitary ensemble for any N with excellent agreement with recent numerical studies. We also discuss the relationship of our findings to models of black hole information loss, such as the