Interface properties might significantly influence the relaxation process of local surface plasmon of noble metal nanostructures, which can modulate their scattering spectral features, hot carrier generation, and optical sensing performance. Here, we investigated the plasmon damping of individual GNRs on the surface of a quartz substrate modified by ion irradiation. The quartz substrates were modified by Ar + ion irradiation with masks to form irradiated and unirradiated regions on the same surface. Then, we investigated the scattering spectra of individual gold nanorods in situ using single-particle spectroscopy and nanomanipulation technique. The plasmonic resonant peaks red-shifted, the linewidths broadened, and the scattering intensities weakened when the nanorods were moved onto the irradiated region. The distance-dependent measurements show that the change rate of the plasmonic resonant peak is more sensitive for the separation, respective to the change rate of the scattering linewidth and intensity. The scattering spectral linewidth became broader with higher irradiation doses, but the plasmonic resonant peaks shifted slightly. The ion irradiation rearranges the atoms on the substrate surface to form surface defects and increase the effective refractive index. Then, the localized surface plasmon coupled with these surface defects results in new relaxation loss channels. This study contributes more understanding to the relationship between the substrate surface and plasmon damping, which is beneficial for developing optoelectronic devices based on localized surface plasmon.