Cosmological gravitational wave backgrounds (CGWBs) are the conglomeration of unresolved gravitational wave signals from early Universe sources, which make them a promising tool for cosmologists. Because gravitons decouple from the cosmic plasma early on, one can consider interactions between gravitons and any particle species that were present in the very early Universe. We show that analogous to the cosmic microwave background, elastic scattering on any cosmological background will induce small distortions in its energy density spectrum. We then quantify the magnitude of these spin-dependent spectral distortions when attributed to the dark matter in the early Universe. Lastly, we give estimates for potentially measurable distortions on CGWBs due to gravitational scattering by primordial black holes.