Optically active phonon modes in ferroelectrics such as potassium titanyl phosphate (KTP) and potassium titanyl arsenate (KTA) in the ~7–20 THz range play an important role in applications of these materials in Raman lasing and terahertz wave generation. Previous studies with picosecond pulse excitation demonstrated that the interaction of pump pulses with phonons can lead to efficient stimulated Raman scattering (SRS) accompanying optical parametric oscillation or amplification processes (OPO/OPA), and to efficient polariton-phonon scattering. In this work, we investigate the behavior of infrared OPAs employing KTP or KTA crystals when pumped with ~800-nm ultrashort pulses of duration comparable to the oscillation period of the optical phonons. We demonstrate that under conditions of coherent impulsive Raman excitation of the phonons, when the effective χ(2) nonlinearity cannot be considered instantaneous, the parametrically amplified waves (most notably, signal) undergo significant spectral modulations leading to an overall redshift of the OPA output. The pump intensity dependence of the redshifted OPA output, the temporal evolution of the parametric gain, as well as the pump spectral modulations suggest the presence of coupling between the nonlinear optical polarizations PNL of the impulsively excited phonons and those of parametrically amplified waves.