High performance neutron detectors sensitive to both thermal and fast neutrons are of great interest to monitor the high neutron flux produced, e.g., by fission and fusion reactors. An obvious requirement for such an application is neutron irradiation hardness. This is why diamond based neutron detectors are currently under test in some of these facilities. In this paper the damaging effects induced in chemical vapor deposition (CVD) diamond based detectors by a neutron fluence of ∼2×1016 neutrons/cm2 have been studied and significant changes in spectroscopic, electrical, and optical properties have been observed. The detectors are fabricated using high quality synthetic CVD single crystal diamond using the p-type/intrinsic/Schottky metal/L6iF layered structure recently proposed by Marinelli et al. [Appl. Phys. Lett. 89, 143509 (2006)], which allows simultaneous detection of thermal and fast neutrons. Neutron radiation hardness up to at least 2×1014 n/cm2 fast (14 MeV) neutron fluence has been confirmed so far [see Pillon et al., (Fusion Eng. Des. 82, 1174 (2007)]. However, at the much higher neutron fluence of ∼2×1016 neutrons/cm2 damage is observed. The detector response to 5.5 MeV A241m α-particles still shows a well resolved α-peak, thus confirming the good radiation hardness of the device but a remarkable degradation and a significant instability with time of charge collection efficiency and energy resolution arise. Symmetric, nearly Ohmic I-V (current-voltage) characteristics have been recorded from the metal/intrinsic/p-doped diamond layered structure, which before neutron irradiation acted as a Schottky barrier diode with a strong rectifying behavior. The nature and the distribution of the radiation induced damage have been deeply examined by means of cathodoluminescence spectroscopy. A more heavily damaged area into the intrinsic diamond at the same position and with the same extension of the L6iF layer has been found, the increased damage being ascribed to the highly ionizing particles produced in the L6iF layer by thermal neutrons through the nuclear reaction L6i(n,α)T.