Background Diet, in particular the Mediterranean diet, has been associated with better cognitive function and less cognitive decline in older populations. Objectives To quantify associations of a healthy diet, defined by adherence to either the Mediterranean diet, the WHO guidelines, or Dutch Health Council dietary guidelines, with cognitive function and cognitive decline from middle age into old age. Methods From the Doetinchem Cohort Study, a large population-based longitudinal study, 3644 participants (51% females) aged 45–75 y at baseline, were included. Global cognitive function, memory, processing speed, and cognitive flexibility were assessed at 5-y time intervals up to 20-y follow-up. Adherence to the Mediterranean diet was measured with the modified Mediterranean Diet Score (mMDS), adherence to the WHO dietary guidelines with the Healthy Diet Indicator (HDI), and adherence to the Dutch Health Council dietary guidelines 2015 with the modified Dutch Healthy Diet 2015 index (mDHD15-index). The scores on the dietary indices were classified in tertiles (low, medium, high adherence). Linear mixed models were used to model level and change in cognitive function by adherence to healthy diets. Results The highest tertiles of the mMDS, HDI, and mDHD15-index were associated with better cognitive function compared with the lowest tertiles (P values <0.01), for instance at age 65 y equal to being 2 y cognitively younger in global cognition. In addition, compared with the lowest tertiles, the highest tertiles of the mMDS, HDI, and mDHD15-index were statistically significantly associated with 6–7% slower global cognitive decline from age 55 to 75 y, but also slower decline in processing speed (for mMDS: 10%; 95% CI: 2, 18%; for mDHD15: 12%; 95% CI: 6, 21%) and cognitive flexibility (for mDHD15: 10%; 95% CI: 4, 18%). Conclusions Healthier dietary habits, determined by higher adherence to dietary guidelines, are associated with better cognitive function and slower cognitive decline with aging from middle age onwards.