The Cattaneo equations with Caputo–Fabrizio fractional derivative are investigated. A compact finite difference scheme of Crank–Nicolson type is presented and analyzed, which is proved to have temporal accuracy of second order and spatial accuracy of fourth order. Since this derivative is defined with an integral over the whole passed time, conventional direct solvers generally take computational complexity of OMN2 and require memory of OMN, with M and N the number of space steps and time steps, respectively. We develop a fast evaluation procedure for the Caputo–Fabrizio fractional derivative, by which the computational cost is reduced to OMN operations; meanwhile, only OM memory is required. In the end, several numerical experiments are carried out to verify the theoretical results and show the applicability of the fast compact difference procedure.