Neuropathological changes associated with Alzheimer’s disease (AD) precede symptom onset by more than a decade. Possession of an Apolipoprotein-E (APOE) ε4 allele is the strongest genetic risk factor for late onset AD. Cross-sectional studies of cognitively intact elders have noted smaller hippocampal/medial temporal volumes in ε4 carriers (ε4+) compared to ε4 non-carriers (ε4-). Few studies, however, have examined long-term, longitudinal, anatomical brain changes comparing healthy ε4+ and ε4- individuals. The current five-year study examined global and regional volumes of cortical and subcortical grey and white matter and ventricular size in 42 ε4+ and 30 ε4- individuals. Cognitively intact participants, ages 65-85 at study entry, underwent repeat anatomical MRI scans on three occasions: baseline, 1.5, and 4.75 years. Results indicated no between group volumetric differences at baseline. Over the follow-up interval, the ε4+ group experienced a greater rate of volume loss in total grey matter, bilateral hippocampi, right hippocampal subfields, bilateral lingual gyri, parahippocampal gyrus, and right lateral orbitofrontal cortex compared to the ε4- group. Greater loss in grey matter volumes in ε4+ participants were accompanied by greater increases in lateral, third and fourth ventricular volumes. Rate of change in white matter volumes did not differentiate the groups. The current results indicate that longitudinal measurements of brain atrophy can serve as a sensitive biomarker for identifying neuropathological changes in persons at genetic risk for AD and potentially, for assessing the efficacy of treatments designed to slow or prevent disease progression during the preclinical stage of AD.