We examined the health of a control group (18–81 years) in our aging study, which is similar to control groups used in other neuroimaging studies. The current study was motivated by our previous results showing that one third of the elder control group had moderate to severe white matter hyperintensities and/or cortical volume loss which correlated with poor performance on memory tasks. Therefore, we predicted that cardiovascular risk factors (e.g., hypertension, high cholesterol) within the control group would account for significant variance on working memory task performance. Fifty-five participants completed 4 verbal and spatial working memory tasks, neuropsychological exams, diffusion tensor imaging (DTI), and blood tests to assess vascular risk. In addition to using a repeated measures ANOVA design, a cluster analysis was applied to the vascular risk measures as a data reduction step to characterize relationships between conjoint risk factors. The cluster groupings were used to predict working memory performance. The results show that higher levels of systolic blood pressure were associated with: 1) poor spatial working memory accuracy; and 2) lower fractional anisotropy (FA) values in multiple brain regions. In contrast, higher levels of total cholesterol corresponded with increased accuracy in verbal working memory. An association between lower FA values and higher cholesterol levels were identified in different brain regions from those associated with systolic blood pressure. The conjoint risk analysis revealed that Risk Cluster Group 3 (the group with the greatest number of risk factors) displayed: 1) the poorest performance on the spatial working memory tasks; 2) the longest reaction times across both spatial and verbal memory tasks; and 3) the lowest FA values across widespread brain regions. Our results confirm that a considerable range of vascular risk factors are present in a typical control group, even in younger individuals, which have robust effects on brain anatomy and function. These results present a new challenge to neuroimaging studies both for defining a cohort from which to characterize `normative' brain circuitry and for establishing a control group to compare with other clinical populations.