The role of vascular risk factors in age-related brain degeneration has long been the subject of intense study. As its own sub-category of vascular risk, obesity has an increasingly recognized role in influencing brain health and health-care strategies, but its association with brain health remains under-studied. Notably, no prior study has addressed sex differences in the association between metrics of obesity and white-matter microstructural integrity, an important early marker of brain degeneration, despite known sex differences in fat storage and usage. This study focuses on the associations between markers of body fat (abdominal fat ratio: AFR, and liver proton density fat fraction: PDFF) and brain microstructural health (measures of white-matter microstructure using diffusion-tensor imaging, DTI). We found that fluid intelligence and reaction time are indeed associated with body fat differently in men and women. We also found significant differences in the associations of AFR with DTI metrics between sexes. These sex differences are mirrored in the associations of SBP and age with DTI metrics. Moreover, these sex differences in the AFR and SBP associations with DTI metrics persist when controlling for age. Taken together, these findings suggest that there are inherent sex-driven differences in how brain health is associated with vascular risk factors such as obesity.