The Spectral Form Factor (SFF) measures the fluctuations in the density of states of a Hamiltonian. We consider a generalization of the SFF called the Loschmidt Spectral Form Factor, $$ \textrm{tr}\left[{e}^{i{H}_1T}\right]\textrm{tr}\left[{e}^{-i{H}_2T}\right] $$
tr
e
i
H
1
T
tr
e
−
i
H
2
T
, for H1 − H2 small. If the ensemble average of the SFF is the variance of the density fluctuations for a single Hamiltonian drawn from the ensemble, the averaged Loschmidt SFF is the covariance for two Hamiltonians drawn from a correlated ensemble. This object is a time-domain version of the parametric correlations studied in the quantum chaos and random matrix literatures. We show analytically that the averaged Loschmidt SFF is proportional to eiλTT for a complex rate λ with a positive imaginary part, showing in a quantitative way that the long-time details of the spectrum are exponentially more sensitive to perturbations than the short-time properties. We calculate λ in a number of cases, including random matrix theory, theories with a single localized defect, and hydrodynamic theories.