Cognitive dysfunction, especially memory impairment, is a typical clinical feature of long-term symptoms caused by repetitive mild traumatic brain injury (rmTBI). The current study aims to investigate the relationship between regional brain atrophy and cognitive impairments in retired athletes with a long history of rmTBI. Overall, 27 retired athletes with a history of rmTBI (18 boxers, 3 kickboxers, 2 wrestlers, and 4 others; rmTBI group) and 23 age/sex-matched healthy participants (control group) were enrolled. MPRAGE on 3T MRI was acquired and segmented. The total brain volume (TBV) and regional brain volumes of the mammillary bodies (MBs), hippocampi, amygdalae, thalami, caudate nuclei, and corpus callosum (CC) were estimated using the SPM12 and ITK–SNAP tools. The rmTBI group underwent neuropsychological tests for verbal and visual memory, attention, executive function, and global cognitive function. TBV and TBV-adjusted regional brain volumes were compared between rmTBI and control participants using the Mann-Whitney U test. Furthermore, we assessed the relationship of the memory test to TBV and TBV-adjusted regional brain volumes by the Spearman’s rank correlation coefficient. The TBV and TBV–adjusted regional brain volumes were compared between groups, and the relationship between the neuropsychological test scores and the regional brain volumes were evaluated. Compared with the control group, the rmTBI group showed significantly lower MBs volume/TBV ratio (0.13 ± 0.05 vs. 0.19 ± 0.03 ×10− 3, p < 0.05) and The CC volume/TBV ratio (12.5 ± 1.63 vs. 13.6 ± 1.81 ×10− 3, p < 0.05). The MBs volume/TBV ratio correlated with visual and verbal memory, as assessed, respectively, by the Rey–Osterrieth Complex Figure test delayed recall (ρ = 0.48, p < 0.05) and logical memory delayed recall (ρ = 0.41, p < 0.05). The CC volume/TBV ratio didn’t correlate with any cognitive assessment result (p > 0.05). MB disconnection from the Papez circuit is associated with memory impairment in retired athletes with rmTBI.