Age-related changes in memory are not uniform, even in the absence of dementia. Characterization of non-disease associated cognitive changes is crucial to gain a more complete understanding of brain aging. Episodic memory was investigated in 13,037 ethnically diverse elderly (ages 72 to 85 years) with two to 15 years of follow-up, and with known dementia status, age, sex, education, and APOE genotypes. Adjusted trajectories of episodic memory performance over time were estimated using Latent Class Mixed Models. Analysis was conducted using two samples at baseline evaluation: i) non-cognitively impaired individuals, and ii) all individuals regardless of dementia status. We calculated the age-specific annual incidence rates of dementia in the non-demented elderly (n = 10,220). Two major episodic memory trajectories were estimated: 1) Stable—consisting of individuals exhibiting a constant or improved memory function, and 2) Decliner—consisting of individuals whose memory function declined. The majority of the study participants maintain their memory performance over time. Compared to those with Stable trajectory, individuals characterized as Decliners were more likely to have non-white ethnic background, fewer years of education, a higher frequency of ε4 allele at APOE gene and five times more likely to develop dementia. The steepest decline in episodic memory was observed in Caribbean-Hispanics compared to non-Hispanic whites (p = 4.3 x 10−15). The highest incident rates of dementia were observed in the oldest age group, among those of Caribbean-Hispanics ancestry and among Decliners who exhibited rates five times higher than those with Stable trajectories (11 per 100 person-years versus 3 per 100 person-years. Age, education, ethnic background and APOE genotype influence the maintenance of episodic memory. Declining memory is one of the strongest predictors of incident dementia.