Brain age can be estimated using different Magnetic Resonance Imaging (MRI) modalities including diffusion MRI. Recent studies demonstrated that white matter (WM) tracts that share the same function might experience similar alterations. Therefore, in this work, we sought to investigate such issue focusing on five WM bundles holding that feature that is Association, Brainstem, Commissural, Limbic and Projection fibers, respectively. For each tract group, we estimated brain age for 15,335 healthy participants from United Kingdom Biobank relying on diffusion MRI data derived endophenotypes, Bayesian ridge regression modeling and 10 fold-cross validation. Furthermore, we estimated brain age for an Ensemble model that gathers all the considered WM bundles. Association analysis was subsequently performed between the estimated brain age delta as resulting from the six models, that is for each tract group as well as for the Ensemble model, and 38 daily life style measures, 14 cardiac risk factors and cardiovascular magnetic resonance imaging features and genetic variants. The Ensemble model that used all tracts from all fiber groups (FG) performed better than other models to estimate brain age. Limbic tracts based model reached the highest accuracy with a Mean Absolute Error (MAE) of 5.08, followed by the Commissural ($$\hbox {MAE}=5.23$$
MAE
=
5.23
), Association ($$\hbox {MAE}=5.24$$
MAE
=
5.24
), and Projection ($$\hbox {MAE}=5.28$$
MAE
=
5.28
) ones. The Brainstem tracts based model was the less accurate achieving a MAE of 5.86. Accordingly, our study suggests that the Limbic tracts experience less brain aging or allows for more accurate estimates compared to other tract groups. Moreover, the results suggest that Limbic tract leads to the largest number of significant associations with daily lifestyle factors than the other tract groups. Lastly, two SNPs were significantly (p value $$< 5\hbox {E}{-}8$$
<
5
E
-
8
) associated with brain age delta in the Projection fibers. Those SNPs are mapped to HIST1H1A and SLC17A3 genes.