We explore the mechanism of transverse momentum broadening of fast quarks propagating in nuclei, using Drell-Yan (DY) transverse momentum distributions measured in the experiment E866 at FermiLab with beams of 800 GeV protons. Our theoretical analysis is based on the color dipole approach in the target rest frame, which has provided a successful phenomenological description of a variety of hadronic reactions. The present application is relevant to the regime of short coherence length (SCL), where the spatial extent of the fluctuations of the projectile responsible for the Drell-Yan reaction is short compared to the internucleon spacing. In this limit, momentum broadening comes from initial state interactions and is described as color filtering, i.e. absorption of large-size dipoles leading to diminished transverse separation and hence enhanced transverse momentum. The predictions we present are in good agreement with the E866 data. The interactions leading to the acquisition of transverse momentum arise from the color-dipole cross section determined previously from deep-inelastic scattering on proton targets. Aside from the determination of the color-dipole cross section, no other phenomenological input is needed to explain the experimental results. The mean-square momentum broadening of dileptons determined in a recent separate analysis of the data is likewise well described by our theory. These results confirm within the model studied that the origin of momentum broadening in DY is the color dipole cross section mediating soft initial state interactions between the parton of the projectile that initiates the reaction and the nucleons of the nucleus, as provided by the color dipole description. Predictions for broadening observables at RHIC are presented.