Objective: This study investigated the relationship between word reading and white matter (WM) integrity within a neuroanatomical-based reading system comparing adult survivors of childhood brain tumors and controls. It was predicted that the association between WM integrity and word reading would be mediated by processing speed, and this indirect effect would be moderated by group. Method: Thirty-seven adult survivors of childhood brain tumor and typically developing adults participated (age M = 24.19 ± 4.51 years, 62% female). DTI Tractography identified the WM tract for 3 of the reading system connections: inferior fronto-occipital fasciculus (IFOF), arcuate fasciculus (AF), and parietotemporal-occipitotemporal connection (PT-OT). Results: Fractional anisotropy values (FA) of the PT-OT tract were significantly correlated with word reading in survivors and controls (r = .45, .58, respectively; p < .05) and IFOF values were associated with reading in survivors only (r = .59, p < .01). Further, the moderated mediated model was significant for PT-OT and IFOF, such that the indirect effect of processing speed was only present for survivors (CI: PT-OT: 2.90, 28.41, IFOF: 2.92, 40.17). Conclusion: Results suggest the tracts emerging from the occipitotemporal area are a critical component of the reading system in adults. The finding that processing speed was the mechanism by which WM was associated with reading in survivors is in alignment with the developmental cascade model. Current findings bolster the existing theory-based models of reading using innovative diffusion tensor imaging and moderated mediation statistical neurodevelopmental model, establishing the role of processing speed and specific WM pathway integrity in word reading skill.