Enzyme replacement therapy has drastically changed prospects of patients with Pompe disease, a progressive metabolic myopathy. As classic infantile patients survive due to treatment, they exhibit progressive white matter abnormalities, while brain involvement in late-onset patients is not fully elucidated. To study the underlying microstructure of white matter, we acquired structural (T1, T2, FLAIR) and diffusion tensor imaging (DTI) of the brain in 12 classic infantile patients (age 5–20 years) and 18 late-onset Pompe patients (age 11–56 years). Structural images were scored according to a rating scale for classic infantile patients. Fractional anisotropy (FA) and mean diffusivity (MD) from classic infantile patients were compared to a reference population, using a Wilcoxon signed-rank, one sample test. Effect sizes (Hedges’ G) were used to compare DTI metrics across different tracts. For late-onset patients, results were compared to (reported) tractography data on normal aging. In classic infantile patients, we found a significant lower FA and higher MD (p < 0.01) compared to the reference population. Large-association fibers were most severely affected. Classic infantile patients with advanced white matter abnormalities on structural MRI showed the largest deviations from the reference population. FA and MD were similar for younger and older late-onset patients in large WM-association fibers. We conclude that, while no deviations from typical neurodevelopment were found in late-onset patients, classic infantile Pompe patients showed quantifiable, substantially altered white matter microstructure, which corresponded with disease stage on structural MRI. DTI holds promise to monitor therapy response in future therapies targeting the brain.