We compare experimental fluctuation electron microscopy (FEM) speckle data with electron diffraction simulations for thin amorphous carbon and silicon samples. We find that the experimental speckle intensity variance is generally more than an order of magnitude lower than kinematical scattering theory predicts for spatially coherent illumination. We hypothesize that decoherence, which randomizes the phase relationship between scattered waves, is responsible for the anomaly. Specifically, displacement decoherence can contribute strongly to speckle suppression, particularly at higher beam energies. Displacement decoherence arises when the local structure is rearranged significantly by interactions with the beam during the exposure. Such motions cause diffraction speckle to twinkle, some of it at observable time scales. We also find that the continuous random network model of amorphous silicon can explain the experimental variance data if displacement decoherence and multiple scattering is included in the modeling. This may resolve the longstanding discrepancy between X-ray and electron diffraction studies of radial distribution functions, and conclusions reached from previous FEM studies. Decoherence likely affects all quantitative electron imaging and diffraction studies. It likely contributes to the so-called Stobbs factor, where high-resolution atomic-column image intensities are anomalously lower than predicted by a similar factor to that observed here.