The hysteretic softening at small dynamic strains (Payne effect)—related to the rolling resistance and viscoelastic losses of tires—was studied as a function of particle size, filler volume fraction, and temperature for carbon black (CB) reinforced uncrosslinked styrene–butadiene rubber (SBR) and a paste-like material composed of CB-filled paraffin oil. The low-strain limit for dynamic storage modulus was found to be remarkably similar for CB-filled oil and the CB-filled SBR. Small-angle X-ray scattering (SAXS) measurements on the simple composites and detailed data analysis confirmed that the aggregate structures and nature of filler branching/networking of carbon black were virtually identical within oil compared to the high molecular weight polymer matrix. The combined dynamic rheology and SAXS results provide clear evidence that the deformation-induced breaking (unjamming) of the filler network—characterized by filler–filler contacts that are percolated throughout the material—is the main cause for the Payne effect. However, the polymer matrix does play a secondary role as demonstrated by a reduction in Payne effect magnitude with increasing temperature for the CB-reinforced rubber, which was not observed to a significant extent for the oil–CB system.