Excitation of coherent optical phonons in solids provides a pathway for ultrafast modulation of light on a sub-ps timescale. Here, we report on efficient 3.6 THz modulation of light reflected from hybrid metal/semiconductor plasmonic crystals caused by lattice vibrations in a few nm thick layer of elemental tellurium. We observe that surface plasmon polaritons contribute significantly to photoinduced formation of this thin layer at the interface between a telluride-based II-VI semiconductor, such as (Cd,Mg)Te or (Cd,Mn)Te, and a one-dimensional gold grating. The change in interface composition is monitored via the excitation and detection of coherent optical tellurium phonons of A1 symmetry by femtosecond laser pulses in a pump-probe experiment. The patterning of a plasmonic grating onto the semiconductor enhances the transient signal which originates from the interface region. This allows monitoring the layer formation and observing the shift of the phonon frequency caused by confinement of the lattice vibrations in the nm-thick segregated layer. Efficient excitation and detection of coherent optical phonons by means of surface plasmon polaritons are evidenced by the dependence of the signal strength on polarization of pump and probe pulses and its spectral distribution.