Even clinically "asymptomatic" carotid stenosis is associated with multidomain cognitive impairment, gray matter (GM) atrophy, and silent lesion. However, the links between them remain unclear. Using structural MRI data, we examined GM asymmetry index (AI) and white matter hyperintensity (WMH) in 24 patients with severe asymptomatic carotid stenosis (SACS), 24 comorbidity-matched controls, and independent samples of 84 elderly controls and 22 young adults. As compared to controls, SACS patients showed worse verbal memories, higher WMH burden, and rightlateralized GM in posterior middle temporal and mouth-somatomotor regions. These clusters extended to pars triangularis, lateral temporal, and cerebellar regions, when compared with young adults. Further, a full-path of WMH burden (X), GM volume (atrophy, M1), AI (asymmetry, M2), and neuropsychological variables (Y) through a serial mediation model was analyzed. This analysis identified that left-dominated GM atrophy and right-lateralized asymmetry in the posterior middle temporal cortex mediated the relationship between WMH burden and recall memory in SACS patients. These results suggest that the unbalanced hemispheric atrophy in the posterior middle temporal cortex is crucial to mediating relationship between WMH burden and verbal recall memories, which may underlie accelerated aging and cognitive deterioration in patients with SACS and other vascular cognitive impairment.