Huntington's disease (HD) is a monogenic neurodegenerative disorder caused by a CAG‐repeat expansion in the Huntingtin gene. Presence of this expansion signifies certainty of disease onset, but only partly explains age at which onset occurs. Genome‐wide association studies have shown that naturally occurring genetic variability influences HD pathogenesis and disease onset. Investigating the influence of biological traits in the normal population, such as variability in white matter properties, on HD pathogenesis could provide a complementary approach to understanding disease modification. We have previously shown that while white matter diffusivity patterns in the left sensorimotor network were similar in controls and HD gene‐carriers, they were more extreme in the HD group. We hypothesized that the influence of natural variation in diffusivity on effects of HD pathogenesis on white matter is not limited to the sensorimotor network but extends to cognitive, limbic, and visual networks. Using tractography, we investigated 32 bilateral pathways within HD‐related networks, including motor, cognitive, and limbic, and examined diffusivity metrics using principal components analysis. We identified three independent patterns of diffusivity common to controls and HD gene‐carriers that predicted HD status. The first pattern involved almost all tracts, the second was limited to sensorimotor tracts, and the third encompassed cognitive network tracts. Each diffusivity pattern was associated with network specific performance. The consistency in diffusivity patterns across both groups coupled with their association with disease status and task performance indicates that naturally‐occurring patterns of diffusivity can become accentuated in the presence of the HD gene mutation to influence clinical brain function.