Disordering of crystal lattice induced by irradiation with fast neutrons and other high-energy particles is used for the deep modification of electrical and optical properties of diamonds via significant nanoscale restructuring and defects engineering. Raman spectroscopy was employed to investigate the nature of radiation damage below the critical graphitization level created when chemical vapor deposition and natural diamonds are irradiated by fast neutrons with fluencies from 1 × 1018 to 3 × 1020 cm−2 and annealed at the 100–1700 °C range. The significant changes in the diamond Raman spectra versus the neutron-irradiated conditions are associated with the formation of intrinsic irradiation-induced defects that do not completely destroy the crystalline feature but decrease the phonon coherence length as the neutron dose increases. It was shown that the Raman spectrum of radiation-damaged diamonds is determined by the phonon confinement effect and that the boson peak is present in the Raman spectra up to annealing at 800–1000 °C. Three groups of defect-induced bands (first group = 260, 495, and 730 cm−1; second group = 230, 500, 530, 685, and 760 cm–1; and third group = 335, 1390, 1415, and 1740 cm−1) were observed in Raman spectra of fast-neutron-irradiated diamonds.