We theoretically study the pump-probe response of nonequilibrium BCS superconductors coupled to optical phonons. For ultrashort pump pulses a nonadiabatic regime emerges, which is characterized by oscillations of the superconducting order parameter as well as by the generation of coherent phonons. Using the density-matrix formalism, we compute the pump-probe response in the nonadiabatic regime of the coupled Bogoliubov quasiparticle-phonon system and determine the signatures of the order parameter and the phonon oscillations in the pump-probe conductivity. We find that the nonadiabatic dynamics of the BCS superconductor reflects itself in oscillations of the pump-probe response as functions of delay time δt between pump and probe pulses. We argue that from the analysis of this oscillatory behavior both frequency and decay time of the algebraically decaying order-parameter oscillations can be inferred. Similarly, the coherent phonons are evidenced in the pump-probe conductivity by oscillations with the frequency of the phonons. Remarkably, we find that the oscillatory response in the pump-probe conductivity is resonantly enhanced when the frequency of the order-parameter oscillations is tuned to the phonon energy.