To relate individual differences in music perception ability with whole brain white matter connectivity, we scanned a group of 27 individuals with varying degrees of musical training and assessed musical ability in sensory and sequential music perception domains using the Profile of Music Perception Skills‐Short version (PROMS‐S). Sequential processing ability was estimated by combining performance on tasks for Melody, Standard Rhythm, Embedded Rhythm, and Accent subscores while sensory processing ability was ascertained via tasks of Tempo, Pitch, Timbre, and Tuning. Controlling for musical training, gender, and years of training, network‐based statistics revealed positive linear associations between total PROMS‐S scores and increased interhemispheric fronto‐temporal and parieto‐frontal white matter connectivity, suggesting a distinct segregated structural network for music perception. Secondary analysis revealed two subnetworks for sequential processing ability, one comprising ventral fronto‐temporal and subcortical regions and the other comprising dorsal fronto‐temporo‐parietal regions. A graph‐theoretic analysis to characterize the structural network revealed a positive association of modularity of the whole brain structural connectome with the d′ total score. In addition, the nodal degree of the right posterior cingulate cortex also showed a significant positive correlation with the total d′ score. Our results suggest that a distinct structural network of connectivity across fronto‐temporal, cerebellar, and cerebro‐subcortical regions is associated with music processing abilities and the right posterior cingulate cortex mediates the connectivity of this network.