The vibrational density of states DðωÞ of solids controls their thermal and transport properties. In crystals, the low-frequency modes are extended phonons distributed in frequency according to Debye's law, DðωÞ ∝ ω 2 . In amorphous solids, phonons are damped, and at low frequency DðωÞ comprises extended modes in excess over Debye's prediction, leading to the so-called boson peak in DðωÞ=ω 2 at ω bp , and quasilocalized ones. Here we show that boson peak and phonon attenuation in the Rayleigh scattering regime are related, as suggested by correlated fluctuating elasticity theory, and that amorphous materials can be described as homogeneous isotropic elastic media punctuated by quasilocalized modes acting as elastic heterogeneities. Our numerical results resolve the conflict between theoretical approaches attributing amorphous solids' vibrational anomalies to elastic disorder and localized defects.