Improving upon the present background rejection capabilities of the cryogenic germanium detectors for direct dark matter search involves an in-depth comprehension of the charge collection process in these devices. Experimental data point to the combined effects of lattice and impurity scattering on the anisotropy of electron transport in Ge at mK temperatures. A Monte Carlo simulation code has been implemented to incorporate these features in a consistent model for charge collection. In a novel approach to carrier scattering by charged impurities in Ge at cryogenic temperatures, the scattering field of the impurities is treated statistically as a random contribution to the collection field, described by the Holtsmark distribution function with a single adjustable parameter, the mean density of the charged centers. Simulation of charge collection along these lines in devices different by their impurity content shows excellent agreement to experiment. Especially noteworthy is the fact that the strength of impurity scattering is reversed from the known concentration of dopant impurities in the crystals, as the crystal with the higher dopant concentration shows lower scattering at low field than the one with the lower concentration. This raises as an issue for further improvement of these devices, the question of the nature of the scattering centers in high-purity Ge crystals at cryogenic temperatures, associated presumably with deep level impurities or crystal defects. Fig. 1 (Color online) (a) Simulated carrier trajectories for a 60 keV energy deposit from a collimated 241 Am γ source facing the bottom surface of the detector crystal. See Ref. [1] for a full description of the device and of the geometry of the charge collection field especially. Detector bias V p = 2 V; T = 20 mK. Holes are in red and electrons in blue, black, green and magenta depending on their valley of belonging. (b) Same as (a), with the polarity of the electrode biases reversed so that holes instead of electrons are drifted across the full thickness of the detector crystal