The impact of quenched disorder on deterministic diffusion in chaotic dynamical systems is studied. As a simple example, we consider piecewise linear maps on the line. In computer simulations we find a complex scenario of multiple suppression and enhancement of normal diffusion, under variation of the perturbation strength. These results are explained by a theoretical approximation, showing that the oscillations emerge as a direct consequence of the unperturbed diffusion coefficient, which is known to be a fractal function of a control parameter.