Cigarette smoking has been associated with dementia and dementia-related brain changes, notably gray matter (GM) volume atrophy. These associations are thought to reflect the co-morbidity of smoking and vascular, respiratory, and substance use/psychological conditions. However, the extent and localization of the smoking-GM relationship and the degree to which vascular, respiratory, and substance use/psychological factors influence this relationship remain unclear. In the Coronary Artery Risk Development in Young Adults CARDIA cohort (n = 698; 52% women; 40% black participants; age = 50.3 (SD = 3.5)), we examined the associations of smoking status with total GM volume and GM volume of brain regions linked to neurocognitive and addiction disorders. Linear regression models were used to adjust for vascular, respiratory, and substance use/psychological factors and to examine whether they modify the smoking-GM relationship. Compared to never-smokers, current smokers had smaller total GM volume (−8.86 cm3 (95%CI = −13.44, −4.29). Adjustment for substance use/psychological – but not vascular or respiratory – factors substantially attenuated this association (coefficients = −5.54 (95% CI = −10.32, −0.76); −8.33 (95% CI = −12.94, −3.72); −7.69 (95% CI = −6.95, −4.21), respectively). There was an interaction between smoking and alcohol use such that among alcohol non-users, smoking was not related to GM volumes and among alcohol users, those who currently smoked had −12 cm3 smaller total GM, specifically in the frontal and temporal lobes, amygdala, cingulate, and insula. Results suggest a large-magnitude association between smoking and smaller GM volume at middle age, accounting for vascular, respiratory, and substance use/psychological factors, and that the association was strongest in alcohol users. Regions suggested to be most vulnerable are those where cognition and addiction processes overlap.