To study the variation tendency of cerebral small vessel disease (CSVD) imaging markers and total burden with aging and to research the relationship between aging, CSVD markers and cognitive function. Methods: Participants in local urban communities were recruited for neuropsychological and magnetic resonance imaging assessments. Montreal Cognitive Assessment (MoCA), Mini-mental State Examination (MMSE), Number Connection Test A (NCT-A) and Digital Symbol Test (DST) were adopted as neuropsychological scale. Age was stratified at 5-year intervals, and the variation tendency of imaging markers and variables of neuropsychological scales in different age groups was studied. We further studied the relationship between aging, image markers and neuropsychological scales by multi-linear regression. Results: Finally, a total of 401 stroke-free participants (age, 54.83±7.74y; 45.9% were male) were included in the present analysis. With the increase of age, the incidence of imaging markers of CSVD were increased with aging except cerebral microbleeds. The performance results of NCT-A and DST were significant difference in 6 age groups (P < 0.001). In addition, linear decline of the neuropsychological function reflected by NCT-A and DST variables was observed. Linear regression found that age was an independent factor affecting the neuropsychological function reflected by NCT-A and DST variables, and the standard correction coefficients among different age groups increased gradually with age. In addition, brain atrophy is an independent factor affecting neuropsychological variables (odds ratio: −2.929, 95% CI: [−5.094 to −0.765]). There was no correlation between the number of neuroimaging markers and neuropsychological variables after full adjustment. Conclusion: There are many CVSD markers even in younger people, the incidence rate and CVSD marker numbers increase with age. Aging and CSVD may eventually affect cognitive function through brain atrophy.