Cognitive frailty (CF) is a clinical condition defined by the presence of both mild cognitive impairment (MCI) and physical frailty (PF). Elderly with CF are at greater risk of dementia than those with MCI or PF alone, but there are few known clinical or neuroimaging features to reliably distinguish CF from PF or MCI. We therefore conducted a population-based cross-sectional study of community elderly combining physical, cognitive, neuropsychiatric, and multisequence magnetic resonance imaging (MRI) evaluations. The MRI evaluation parameters included white matter (WM) lesion volumes, perivascular and deep subcortical WM lesion grades, lacunar infarct prevalence, microbleed number, and regional medial temporal lobe (MTL) volumes. Participants were divided into 4 groups according to the presence or absence of MCI and PF—(1) no MCI, PF (n = 27); (2) no PF, MCI (n = 119); (3) CF (MCI + PF) (n = 21), (4) normal controls (n = 716). Unique features of CF included shorter one-leg standing time; severe depressive symptoms; and MRI signs of significantly more WM lesions, lacunar infarcts, small-vessel disease lesions, microbleeds, and reduced MTL volumes. These unique deficits suggest that interventions for CF prevention and treatment should focus on motor skills, depressive symptoms, and vascular disease risk factor control.