Background: The long-term (>12 months) prevalence, predictors, and trajectory of post-stroke cognitive deficits are not well established, especially at a community level. This study investigated the longitudinal course and prevalence of cognitive impairment in an incidence cohort, identifying factors associated with declining cognition. Methods: Two hundred fifty-seven participants (mean age = 67.93 ± 13.59) of first-ever stroke survivors, completed cognitive assessments within 2 weeks post stroke, and/or 1, 6, 12, and 48-month. Multivariate linear and logistic models were used to identify baseline predictors (reported as OR with 95% CI) and trajectory of cognitive impairment. Results: Cognitive functioning significantly declined by 2.8 points by 4 years post stroke. Eighty-four percent of stroke survivors had cognitive impairment indicative of post-stroke dementia (mean Montreal cognitive assessment = 20 ± 4.7) at 4-year. There were significant associations between progressive cognitive decline and the following factors: male gender (OR 2.9, 95% CI 1.6–5.9, p = 0.0171), coronary artery disease (OR 2.96, 95% CI 1.35–6.49, p = 0.0070), arrhythmia (OR 2.21, 95% CI 1.07–4.57, p = 0.0317), not in a relationship (OR 2.8, 95% CI 1.4–5.50, p < 0.0001), and not employed (OR 4.9, 95% CI 1.9–12.1, p < 0.0001). Conclusions: Cognitive deficits remain highly prevalent at 4-year post stroke. Early identification of those at higher risk of declining cognition is vital to target rehabilitation interventions at the acute stage and improve overall outcomes.