We present a general model, based on a Hamiltonian approach, for the joint quantum state of photon pairs generated through pulsed spontaneous four-wave mixing, including nonlinear phase modulation and a finite material response time. For the case of a silica fiber, it is found that the pair-production rate depends weakly on the waveguide temperature, due to higher-order Raman scattering events, and more strongly on pump-pair frequency detuning. From the analytical model, a numerical scheme is derived, based on the well-known split-step method. This scheme allows computation of joint states where nontrivial effects are included, such as groupvelocity dispersion and Raman scattering. In this work, the numerical model is used to study the impact of the noninstantaneous response on the prefiltering purity of heralded single photons. We find that for pump pulses shorter than 1 ps, a significant detuning-dependent change in quantum-mechanical purity may be observed in silica. This shows that Raman scattering not only introduces noise, but can also drastically change the spectral correlations in photon pairs when pumped with short pulses.