SummaryConcerns have been raised about the effects on cognition of anaesthesia for surgery, especially in elderly people. We recorded cognitive decline in a cohort of 394 people (198 women) with median (IQR) age at recruitment of 72.6 (66.6–77.8) years, of whom 109 had moderate or major surgery during a median (IQR) follow‐up of 4.1 (2.0–7.6) years. Cognitive decline was more rapid in people who on recruitment were: older, p = 0.0003; male, p = 0.027; had worse cognition, p < 0.0001; or carried the ε4 allele of apoliprotein E (APOEε4), p = 0.008; and after an operation if cognitive impairment was already diagnosed, p = 0.0001. Cognitive decline appears to accelerate after surgery in elderly patients diagnosed with cognitive impairment, but not other elderly patients.