Structural imaging studies suggest gender differences in brain volumes but whether hormone treatment (HT) can protect against age-related structural changes remains unknown and no prior neuroimaging study has investigated potential interactions between HT and estrogen receptor (ESR) polymorphisms. MRI was used to measure grey and white matter, hippocampal volume, corpus callosum, cerebrospinal fluid (CSF), total intracranial volume (ICV) and white matter lesions (WML) in 582 non-demented older adults. In multivariable analysis when compared to women never using HT, men and women currently on treatment, but not past users, had significantly smaller ratio of grey matter to ICV and increased atrophy (CSF/ICV ratio). Hippocampal and white matter volume, and corpus callosum area were not significantly different across groups. ESR2 variants were not significantly associated with brain measures, but women with the ESR1 rs2234693 C allele had significantly smaller WML. Furthermore this association was modified by HT use. Our results do not support a beneficial effect of HT on brain volumes in older women but suggest the potential involvement of ESR1 in WML.