Cerebral small vessel disease (CSVD) is a common cause of morbidity and cognitive decline in the elderly population. However, characterizing the disease pathophysiology and its association with potential clinical sequelae in early stages is less well explored. We applied fixel-based analysis (FBA), a novel framework of investigating microstructural white matter integrity by diffusion-weighted imaging, to data of 921 participants of the Hamburg City Health Study, comprising middle-aged individuals with increased cerebrovascular risk in early stages of CSVD. In individuals in the highest quartile of white matter hyperintensity loads (n=232, median age 63 years; IQR 15.3 years), FBA detected significantly reduced axonal density and increased atrophy of transcallosal fiber tracts, the bilateral superior longitudinal fasciculus, and corticospinal tracts compared to participants in the lowest quartile of white matter hyperintensities (n=228, mean age 55 years; IQR 10 years). Analysis of all participants (N=921) demonstrated a significant association between reduced fiber density and worse executive functions operationalized by the Trail Making Test. Findings were confirmed by complementary analysis of diffusion tensor metrics.