Exploring the use of single-mode fibers (SMFs) in high precision Doppler spectrometers has become increasingly attractive since the advent of diffraction-limited adaptive optics systems on largeaperture telescopes. Spectrometers fed with these fibers can be made significantly smaller than typical 'seeing-limited' instruments, greatly reducing cost and overall complexity. Importantly, classical mode interference and speckle issues associated with multi-mode fibers, also known as 'modal noise', are mitigated when using SMFs, which also provide perfect radial and azimuthal image scrambling. However, these fibers do support multiple polarization modes, an issue that is generally ignored for larger-core fibers given the large number of propagation modes. Since diffraction gratings used in most high resolution astronomical instruments have dispersive properties that are sensitive to incident polarization changes, any birefringence variations in the fiber can cause variations in the efficiency profile, degrading illumination stability. Here we present a cautionary note outlining how the polarization properties of SMFs can affect the radial velocity measurement precision of high resolution spectrographs. This work is immediately relevant to the rapidly expanding field of diffraction-limited, extreme precision RV spectrographs that are currently being designed and built by a number of groups.