Background/Objectives Dietary habits are known to affect health, including the rate of brain ageing and susceptibility to diseases. This study examines the longitudinal relationship between dietary diversity and hippocampal volume, which is a key structure of memory processing and is known to be impaired in dementia. Subjects/Methods Subjects were aged 40-89 years (n = 1683, men: 50.6%) and participated in a 2-year follow-up study of the National Institute for Longevity Sciences-Longitudinal Study of Aging. Dietary intake was calculated from 3-day dietary records, and dietary diversity was determined using the Quantitative Index for Dietary Diversity at baseline. Longitudinal changes in hippocampal and total grey matter volumes were estimated by T1-weighted brain magnetic resonance imaging and FreeSurfer software. Estimated mean brain volume change in relation to dietary diversity score quintiles was assessed by the general linear model, adjusted for age, sex, education, smoking status, alcohol intake, physical activity, and comorbidities. Results The mean (± standard deviation) % decreases in hippocampal and total grey matter volume during the 2-year followup were 1.00% (±2.27%) and 0.78% (±1.83%), respectively. Multivariate-adjusted decreases in total grey matter volume were associated with dietary diversity score (p = 0.065, p for trend = 0.017), and the % decrease in hippocampal volume was more strongly associated with the dietary diversity score: the estimated mean (± standard error) values were 1.31% (±0.12%), 1.07% (±0.12%), 0.98% (±0.12%), 0.81% (±0.12%), and 0.85% (±0.12%), according to dietary diversity quintiles in ascending order (p = 0.030, p for trend = 0.003). Conclusions Among community dwellers, increased dietary diversity may be a new nutritional strategy to prevent hippocampal atrophy.