It is known that in the presence of surface roughness, adhesion can lead to distinct paths of loading and unloading for the area–load and penetration–load relationships, thus causing hysteretic loss. Here, we investigate the effects that the surface roughness parameters have on such adhesive hysteresis loss. We focus on the frictionless normal contact between soft elastic bodies and, for this reason, we model adhesion according to Johnson, Kendall, and Roberts (JKR) theory. Hysteretic energy loss is found to increase linearly with the true area of contact, while the detachment force is negligibly influenced by the maximum applied load reached at the end of the loading phase. Moreover, for the micrometric roughness amplitude hrms considered in the present work, adhesion hysteresis is found to be affected by the shorter wavelengths of roughness. Specifically, hysteresis losses decrease with increasing fractal dimension and cut-off frequency of the roughness spectrum. However, we stress that a different behavior could occur in other ranges of roughness amplitude.