Physical frailty is an age-related clinical syndrome that is associated with multiple adverse health outcomes, including cognitive impairment and dementia. Recent studies have shown that frailty is associated with specific volumetric neuroimaging characteristics. Whether brain microstructural characteristics, particularly gray matter, associated with frailty exist and what their spatial distribution is have not been explored. We identified 670 participants of the Baltimore Longitudinal Study of Aging who were aged 60 and older and cognitively normal and who had concurrent data on frailty and regional microstructural neuroimaging markers by diffusion tensor imaging (DTI), including mean diffusivity (MD) of gray matter and fractional anisotropy (FA) of white matter. We identified neuroimaging markers that were associated with frailty status (non-frail, pre-frail, frail) and further examined differences between three groups using multivariate linear regression (non-frail = reference). Models were adjusted for age, sex, race, years of education, body mass index, scanner type, and Apolipoprotein E e4 carrier status. Compared to the non-frail participants, those who were frail had higher MD in the medial frontal cortex, several subcortical regions (putamen, caudate, thalamus), anterior cingulate cortex, and a trend of lower FA in the body of the corpus callosum. Those who were pre-frail also had higher MD in the putamen and a trend of lower FA in the body of the corpus callosum. Our study demonstrates for the first time that the microstructure of both gray and white matter differs by frailty status in cognitively normal older adults. Brain areas were not widespread but mostly localized in frontal and subcortical motor areas and the body of the corpus callosum. Whether changes in brain microstructure precede future frailty development warrants further investigation.