We study, with numerical simulations using the generalized nonlinear envelope equation, the processes of optical parametric and difference-and sum-frequency generation (SFG) with incoherent pumps in optical media with both quadratic and third-order nonlinearity, such as periodically poled lithium niobate. With ultrabroadband amplified spontaneous emission pumps or continua (spectral widths >10 THz), group-velocity matching of a near-IR pump and a short-wavelength mid-IR (MIR) idler in optical parametric generation may lead to more than 15-fold relative spectral narrowing of the generated MIR signal. Moreover, the SFG process may also lead to 6-fold signal coherence improvements. When using relatively narrowband filtered noise pumps (e.g., spectral widths < 1 THz), the signal from optical parametric, sum-frequency, and difference-frequency generation has nearly the same spectral width as that of the incoherent pump.