Single-atom impurities and other atomic-scale defects can notably alter the local vibrational responses of solids and, ultimately, their macroscopic properties. Using high-resolution electron energy-loss spectroscopy in the electron microscope, we show that a single substitutional silicon impurity in graphene induces a characteristic, localized modification of the vibrational response. Extensive ab initio calculations reveal that the measured spectroscopic signature arises from defect-induced pseudo-localized phonon modes—that is, resonant states resulting from the hybridization of the defect modes and the bulk continuum—with energies that can be directly matched to the experiments. This finding realizes the promise of vibrational spectroscopy in the electron microscope with single-atom sensitivity and has broad implications across the fields of physics, chemistry, and materials science.