We study the heat transfer in weakly interacting particle systems in vacuum. The particles have surface roughness with self-affine fractal properties, as expected for mineral particles produced by fracture, e.g., by crunching brittle materials in a mortar, or from thermal fatigue or the impact of micrometeorites on asteroids. We show that the propagating electromagnetic (EM) waves give the dominant heat transfer for large particles, while for small particles both the evanescent EM-waves and the phononic contribution from the area of real contact are important. As an application, we discuss the heat transfer in rubble pile asteroids.