We perform a numerical investigation of the effect of the disorder associated with randomly located impurities on shot noise in mesoscopic cavities. We show that such a disorder becomes dominant in determining the noise behavior when the amplitude of the potential fluctuations is comparable to the value of the Fermi energy and for a large enough density of impurities. In contrast to existing conjectures, random potential fluctuations are shown not to contribute to achieving the chaotic regime whose signature is a Fano factor of 1/4, but, rather, to the diffusive behavior typical of disordered conductors. In particular, the 1/4 suppression factor expected for a symmetric cavity can be achieved only in high-quality material, with a very low density of impurities. As the disorder strength is increased, a relatively rapid transition of the suppression factor from 1/4 to values typical of diffusive or quasi-diffusive transport is observed. Finally, on the basis of a comparison between a hard-wall and a realistic model of the cavity, we conclude that the specific details of the confinement potential have a minor influence on noise. C