Background Midlife obesity is related to late-onset dementia. Studying adipose tissue-secreted adipokines in the context of brain aging may help us understand this association. Objective To investigate associations between adipokines and brain MRI markers in middle-aged adults from the Third-Generation cohort of the Framingham Heart Study. Methods Serum adiponectin, retinol-binding protein 4 (RBP4), fetuin-A, and Fatty Acid-Binding Protein 4 (FABP4) concentrations were measured by enzymatic immunoassays. MRI measures included total brain, cortical gray matter, hippocampal (total and anterior), lateral ventricular, and white matter hyperintensity volumes. We used linear regression models to separately relate adipokine concentrations to MRI measures, adjusting for potential confounders. Results We included 1882 participants (mean age of 48 ± 8 years, 54% women). Higher RBP4 concentrations were related to markers of brain atrophy, including smaller total (Beta ± standard error, −0.05 ± 0.02; p = 0.014) and cortical gray brain volumes (−0.06 ± 0.02; p = 0.004), and larger lateral ventricular volumes (0.06 ± 0.02; p = 0.006). Additionally, higher RBP4 (−0.06 ± 0.03; p = 0.042), Fetuin-A (−0.06 ± 0.03; p = 0.039), and FABP4 (−0.09 ± 0.03; p = 0.008) concentrations were associated with smaller anterior hippocampal volumes. Most associations remained after additional adjustment for vascular risk factors. In exploratory analyses, higher FABP4 was related to larger total brain in non-obese participants, and to smaller anterior hippocampal volumes in obese participants. Finally, higher adiponectin concentrations were related to smaller cortical gray, only in non-obese participants. Conclusions Our results suggest that adipokines are associated with markers of brain atrophy during midlife. Further studies are needed to replicate these findings and elucidate any potential mechanisms contributing to abnormal brain aging.