Focal brain lesions, such as stroke and tumors, can lead to remote structural alterations across the whole-brain networks. Brain arteriovenous malformations (AVMs), usually presumed to be congenital, often result in tissue degeneration and functional displacement of the perifocal areas, but it remains unclear whether AVMs may produce long-range effects upon the whole-brain white matter organization. In this study, we used diffusion tensor imaging and graph theory methods to investigate the alterations of brain structural networks in 14 patients with AVMs in the presumed Broca’s area, compared to 27 normal controls. Weighted brain structural networks were constructed based on deterministic tractography. We compared the topological properties and network connectivity between patients and normal controls. Functional magnetic resonance imaging revealed contralateral reorganization of Broca’s area in five (35.7%) patients. Compared to normal controls, the patients exhibited preserved small-worldness of brain structural networks. However, AVM patients exhibited significantly decreased global efficiency (
p
=
0.004
) and clustering coefficient (
p
=
0.014
), along with decreased corresponding nodal properties in some remote brain regions (
p
<
0.05
, family-wise error corrected). Furthermore, structural connectivity was reduced in the right perisylvian regions but enhanced in the perifocal areas (
p
<
0.05
). The vulnerability of the left supramarginal gyrus was significantly increased (
p
=
0.039
, corrected), and the bilateral putamina were added as hubs in the AVM patients. These alterations provide evidence for the long-range effects of AVMs on brain white matter networks. Our preliminary findings contribute extra insights into the understanding of brain plasticity and pathological state in patients with AVMs.