This article is dedicated to the following class of problems. Start with an N × N Hermitian matrix randomly picked from a matrix ensemble -the reference matrix. Applying a rank-t perturbation to it, with t taking the values 1 ≤ t ≤ N , we study the difference between the spectra of the perturbed and the reference matrices as a function of t and its dependence on the underlying universality class of the random matrix ensemble. We consider both, the weaker kind of perturbation which either permutes or randomizes t diagonal elements and a stronger perturbation randomizing successively t rows and columns. In the first case we derive universal expressions in the scaled parameter τ = t/N for the expectation of the variance of the spectral shift functions, choosing as random-matrix ensembles Dyson's three Gaussian ensembles. In the second case we find an additional dependence on the matrix size N .