Spectrographs nominally contain a degree of quasi-static optical aberrations resulting from the quality of manufactured component surfaces, imperfect alignment, design residuals, thermal effects, and other other associated phenomena involved in the design and construction process. Aberrations that change over time can mimic the line centroid motion of a Doppler shift, introducing radial velocity (RV) uncertainty that increases time-series variability. Even when instrument drifts are tracked using a precise wavelength calibration source, barycentric motion of the Earth leads to a wavelength shift of stellar light causing a translation of the spectrum across the focal plane array by many pixels. The wavelength shift allows absorption lines to experience different optical propagation paths and aberrations over observing epochs. We use physical optics propagation simulations to study the impact of aberrations on precise Doppler measurements made by diffraction-limited, high-resolution spectrographs. Using the optical model of the iLocater spectrograph, we quantify the uncertainties that cross-correlation techniques introduce in the presence of aberrations and barycentric RV shifts. We find that aberrations which shift the point-spread-function (PSF) photo-center in the dispersion direction, in particular primary horizontal coma and trefoil, are the most concerning. To maintain aberration-induced RV errors less than 10 cm/s, phase errors for these particular aberrations must be held well below 0.05 waves at the instrument operating wavelength. Our simulations further show that wavelength calibration only partially compensates for instrumental drifts, owing to a behavioural difference between how crosscorrelation techniques handle aberrations between starlight versus calibration light. Identifying subtle physical effects that influence RV errors will help ensure that diffraction-limited planet-finding spectrographs are able to reach their full scientific potential.