Chronic alcohol use is associated with declines in gray matter volume, as is the normal aging process. Less apparent, however, is how the interaction between aging and heavy alcohol use affects changes in gray matter across the lifespan. There is some evidence that women are more vulnerable to the negative effects of alcohol use on gray matter than men. In the current study, we examined whether localized gray matter was related to measures of alcohol use disorder (e.g., AUDIT score) in a large sample (N = 436) of participants, ages 18–55 years, with a range of disease severity, using both voxel-based morphometry (VBM) and surface-based morphometry (SBM). We also explored whether gray matter associations with alcohol use disorder (AUD) severity are moderated by sex and age. Results showed significant negative associations between AUD severity and gray matter volume throughout temporal, parietal, frontal, and occipital lobes. Women showed more negative effects of alcohol use than men for cortical thickness in left orbitofrontal cortex, but evidence for increased vulnerability based on sex was limited overall. Similarly, a specific age by alcohol use interaction was observed for volume of right insula, but other regional or global interactions were not statistically supported. However, significant negative associations between heavy alcohol use and gray matter volumes were observed as early as 18–25 years. These findings support that alcohol has deleterious effects on global and regional gray matter above and beyond age, and, of particular importance, that regional associations emerge in early adulthood.