The strongest genetic risk factor for Alzheimer’s disease (AD) is the ε4 allele of Apolipoprotein E (APOE) and recent genome-wide association meta-analyses have confirmed additional associated genetic loci with smaller effects. The aim of this study was to investigate the ability of an AD polygenic risk score (PRS) and APOE status to predict clinical diagnosis of AD, vascular (VD), mixed (MD), and all-cause dementia in a community-based cohort prospectively followed over 17 years and secondarily across age, sex, and education strata. A PRS encompassing genetic variants reaching genome-wide significant associations to AD (excluding APOE) from the most recent genome-wide association meta-analysis data was calculated and APOE status was determined in 5203 participants. During follow-up, 103, 111, 58, and 359 participants were diagnosed with AD, VD, MD, and all-cause dementia, respectively. Prediction ability of AD, VD, MD, and all-cause dementia by the PRS and APOE was assessed by multiple logistic regression and receiver operating characteristic curve analyses. The PRS per standard deviation increase in score and APOE4 positivity (≥1 ε4 allele) were significantly associated with greater odds of AD (OR, 95% CI: PRS: 1.70, 1.45–1.99; APOE4: 3.34, 2.24–4.99) and AD prediction accuracy was significantly improved when adding the PRS to a base model of age, sex, and education (ASE) (c-statistics: ASE, 0.772; ASE + PRS, 0.810). The PRS enriched the ability of APOE to discern AD with stronger associations than to VD, MD, or all-cause dementia in a prospective community-based cohort.