INTRODUCTIONWe estimated the ages when associations between Alzheimer's disease (AD) genes and brain volumes begin among middle‐aged and older adults.METHODSAmong 45,616 dementia‐free participants aged 45–80, linear regressions tested whether genetic risk score for AD (AD‐GRS) had age‐dependent associations with 38 regional brain magnetic resonance imaging volumes. Models were adjusted for sex, assessment center, genetic ancestry, and intracranial volume.RESULTSAD‐GRS modified the estimated effect of age (per decade) on the amygdala (−0.41 mm3 [−0.42, −0.40]); hippocampus (−0.45 mm3 [−0.45, −0.44]), nucleus accumbens (−0.55 mm3 [−0.56, −0.54]), thalamus (−0.38 mm3 [−0.39, −0.37]), and medial orbitofrontal cortex (−0.23 mm3 [−0.24, −0.22]). Trends began by age 45 for the nucleus accumbens and thalamus, 48 for the hippocampus, 51 for the amygdala, and 53 for the medial orbitofrontal cortex. An AD‐GRS excluding apolipoprotein E (APOE) was additionally associated with entorhinal and middle temporal cortices.DISCUSSIONAPOE and other genes that increase AD risk predict lower hippocampal and other brain volumes by middle age.