Aerobic fitness and exercise could preserve white matter (WM) integrity in older adults. This study investigated the effect on WM microstructural organization of 5 years of exercise intervention with either supervised moderate-intensity continuous training (MICT), high-intensity interval training (HIIT), or following the national physical activity guidelines. A total of 105 participants (70–77 years at baseline), participating in the randomized controlled trial Generation 100 Study, volunteered to take part in this longitudinal 3T magnetic resonance imaging (MRI) study. The HIIT group (n = 33) exercised for four intervals of 4 min at 90% of peak heart rate two times a week, the MICT group (n = 24) exercised continuously for 50 min at 70% peak heart rate two times a week, and the control group (n = 48) followed the national guidelines of ≥30 min of physical activity almost every day. At baseline and at 1-, 3-, and 5-year follow-ups, diffusion tensor imaging (DTI) scans were performed, cardiorespiratory fitness (CRF) was measured as peak oxygen uptake (VO2peak) with ergospirometry, and information on exercise habits was collected. There was no group*time or group effect on any of the DTI indices at any time point during the intervention. Across all groups, CRF was positively associated with fractional anisotropy (FA) and axial diffusivity (AxD) at the follow-ups, and the effect became smaller with time. Exercise intensity was associated with mean diffusivity (MD)/FA, with the greatest effect at 1-year and no effect at 5-year follow-up. There was an association between exercise duration and FA and radial diffusivity (RD) only after 1 year. Despite the lack of group*time interaction or group effect, both higher CRF and exercise intensity was associated with better WM microstructural organization throughout the intervention, but the effect became attenuated over time. Different aspects of exercising affected the WM metrics and WM tracts differently with the greatest and most overlapping effects in the corpus callosum. The current study indicates not only that high CRF and exercise intensity are associated with WM microstructural organization in aging but also that exercise’s positive effects on WM may decline with increasing age.