We investigated the IR absorption spectra of CVD diamond damaged by fast neutrons (>0.1 MeV) with high fluences ranging from 1 × 1018 to 2 × 1019 cm−2 and annealed at temperatures of 200 °C to 1680 °C. After annealing above 1000 °C, the formation of “amber-centers” (ACs), associated with multivacancy clusters, is detected as deduced from the appearance of a strong absorption line at 4100 cm−1. Moreover, the concentration of the ACs in the irradiated diamond can be an order of magnitude higher than that observed previously in the darkest brown natural diamonds. A number of other absorption lines, including the H1b center at 4936 cm−1 (0.612 eV) and new lines at ~5700 cm−1 (0.706 eV) and 9320 cm−1 (1.155 eV) not reported before in the literature, are observed, and their intensity evolutions at annealing temperatures are documented. At the highest fluences, all the lines show reduced intensities and broadening and spectral shifts due to a very high defect concentration and partial amorphization. The obtained experimental data can be used for the analysis of defect generation, transformations and healing in irradiated synthetic and natural diamonds.