The apolipoprotein E epsilon 4 allele (ApoE-ε4) is the strongest known genetic risk factor for late onset Alzheimer’s disease. Expansion of the lateral ventricles occurs with normal aging, but dementia accelerates this process. Brain structure and function depend on ApoE genotype not just for AD patients but also in healthy elderly, and even in asymptomatic young individuals. Therefore, we hypothesized that the ApoE-ε4 allele is associated with altered patterns of longitudinal ventricular expansion, in dementia and normal aging. We tested this hypothesis in a large sample of elderly participants, using a Linear Discriminant Analysis-based approach. Carrying more ApoE-ε4 alleles was associated with faster ventricular expansion bilaterally and with regional patterns of lateral ventricle morphology at 1- and 2-year follow up, after controlling for sex, age, and dementia status. ApoE genotyping is considered critical in clinical trials of AD. These findings, combined with earlier investigations showing that ApoE is also directly implicated in other conditions, suggest that the selective enrollment of ApoE-ε4 carriers may empower clinical trials of other neurological disorders.