Background: Cigarette smoking is believed to accelerate age-related neurodegeneration. Despite significant sex differences in both smoking behaviors and brain structures, the active literature is equivocal in parsing out a sex difference in smoking-associated brain structural changes. Objective: The current study examined subcortical and lateral ventricle gray matter (GM) volume differences among smokers, active, past, and never-smokers, stratified by sex. Methods: The current study data included 1959 Dallas Heart Study (DHS) participants with valid brain imaging data. Stratified by gender, multiple-group comparisons of three cigarette-smoking groups were conducted to test whether there is any cigarette-smoking group differences in GM volumes of the selected regions of interest (ROIs). Results: The largest subcortical GM volumetric loss and enlargement of the lateral ventricle were observed among past smokers for both females and males. However, these observed group differences in GM volumetric changes were statistically significant only among males after adjusting for age and intracranial volumes. Conclusions: The study findings suggest a sex difference in lifetime-smoking-associated GM volumetric changes, even after controlling for aging and intracranial volumes.