Brain gray- and white matter changes is well described in alcohol-dependent elderly subjects; however, the effect of lower levels of alcohol consumption on the brain is poorly understood. We investigated the impact of different amounts of weekly alcohol consumption on brain structure in a population-based sample of 70-year-olds living in Gothenburg, Sweden. Cross-sectional data from 676 participants from The Gothenburg H70 Birth Cohort Study 2014–16 were included. Current alcohol consumers were divided into seven groups based on self-reported weekly amounts of alcohol consumption in grams (g) (0–50 g/week, used as reference group, 51–100 g/week, 101–150 g/week, 151–200 g/week, 201–250 g/week, 251–300 g/week, and > 300 g/week). Subcortical volumes and cortical thickness were assessed on T1-weighted structural magnetic resonance images using FreeSurfer 5.3, and white matter integrity assessed on diffusion tensor images, using tract-based statistics in FSL. General linear models were carried out to estimate associations between alcohol consumption and gray- and white matter changes in the brain. Self-reported consumption above 250 g/week was associated with thinning in the bilateral superior frontal gyrus, the right precentral gyrus, and the right lateral occipital cortex, in addition to reduced fractional anisotropy (FA) and increased mean diffusivity (MD) diffusively spread in many tracts all over the brain. No changes were found in subcortical gray matter structures. These results suggest that there is a non-linear relationship between alcohol consumption and structural brain changes, in which loss of cortical thickness only occur in non-demented 70-year-olds who consume more than 250 g/week.