Aging is associated with increased carotid artery stiffness, a predictor of incident stroke, and reduced cognitive performance and brain white matter integrity (WMI) in humans. Therefore, we hypothesized that higher carotid stiffness/lower compliance would be independently associated with slower processing speed, higher working memory cost, and lower WMI in healthy middle-aged/older (MA/O) adults. Carotid β-stiffness ( < 0.001) was greater and compliance ( < 0.001) was lower in MA/O ( = 32; 64.4 ± 4.3 yr) vs. young ( = 19; 23.8 ± 2.9 yr) adults. MA/O adults demonstrated slower processing speed (27.4 ± 4.6 vs. 35.4 ± 5.0 U/60 s, < 0.001) and higher working memory cost (-15.4 ± 0.14 vs. -2.2 ± 0.05%, < 0.001) vs. young adults. Global WMI was lower in MA/O adults ( < 0.001) and regionally in the frontal lobe ( = 0.020) and genu ( = 0.009). In the entire cohort, multiple regression analysis that included education, sex, and body mass index, carotid β-stiffness index (B = -0.53 ± 0.15 U, = 0.001) and age group (B = -4.61 ± 1.7, = 0.012, adjusted = 0.4) predicted processing speed but not working memory cost or WMI. Among MA/O adults, higher β-stiffness (B = -0.60 ± 0.18, = 0.002) and lower compliance (B = 0.93 ± 0.26, = 0.002) were associated with slower processing speed but not working memory cost or WMI. These data suggest that greater carotid artery stiffness is independently and selectively associated with slower processing speed but not working memory among MA/O adults. Carotid artery stiffening may modulate reductions in processing speed earlier than working memory with healthy aging in humans. Previously, studies investigating the relation between large elastic artery stiffness, cognition, and brain structure have focused mainly on aortic stiffness in aged individuals with cardiovascular disease risk factors and other comorbidities. This study adds to the field by demonstrating that the age-related increases in carotid artery stiffness, but not aortic stiffness, is independently and selectively associated with slower processing speed but not working memory among middle-aged/older adults with low cardiovascular disease risk factor burden.