<b><i>Introduction:</i></b> The neuropsychological feature of vascular mild cognitive impairment is a deficit of the frontal-subcortical circuit; however, the features in the early stage are not consistent. In the present study, we aimed to investigate the neuropsychological features of the very early stage of cognitive impairment with cerebral small vessel disease (CSVD) and to elucidate the cognitive differences among CSVD subtypes. <b><i>Methods:</i></b> A comprehensive neuropsychological test battery was applied to nondemented subjects scoring below the cutoff point 26 of the Japanese version of the Montreal Cognitive Assessment. After factor analysis was conducted to identify covert cognitive factors in the battery, correlation analyses were performed between the factors and CSVD subtypes: white matter hyperintensity (WMH), lacunar infarcts (LIs), cerebral microbleeds (CMBs), perivascular spaces, and cortical atrophy. <b><i>Results:</i></b> Among the 465 recruited patients, 139 underwent a full neuropsychological test battery. Through factor analysis, the following three factors were extracted: executive function, memory, and attention. Of the CSVD features, total WMH was correlated with executive function and memory, whereas deep WMH was correlated with memory alone. Of the CSVD subtypes, LIs and CMBs were correlated only with executive function. Frontal and posterior atrophy were correlated with memory and attention, whereas medial temporal atrophy was correlated with memory alone. <b><i>Conclusions:</i></b> Executive dysfunction accompanied by subtle impairment of memory and processing speed was the main feature of neuropsychological profiles in the subjects with CSVD, even in the very early stage. Furthermore, each CSVD feature and focal cerebral atrophy are associated with cognitive impairment.