Olfactory function has significant implications for human health, but few risk factors for olfactory decline have been identified. We examined the factors associated with olfactory status and decline over five years in the Atherosclerosis Risk in Communities (ARIC) Neurocognitive Study. A 12-item odor identification test was used to assess olfaction in 6053 participants in 2011–2013 (ARIC visit 5, mean age: 75.6, 41% male, 23% Black race) and in 3235 participants in 2016–2017 (visit 6). We used Poisson regression models to examine cross-sectional associations of a range of potential factors with the total odor identification errors (mean errors: 2.8 ± 2.4) in visit 5 participants. We used mixed-effect Poisson regression to examine associations with olfactory decline between visits 5 and 6. We also examined associations with visit 5 anosmia prevalence (847 cases, 14%) and incident anosmia between the two visits (510 cases, 16%) using Poisson models. Older age, male sex, lower education, Black race, APOE ε4 alleles, and diabetes were associated with higher odor identification errors and higher anosmia prevalence, and greater physical activity and hypertension with better olfaction. Age, male sex, lower education, Black race, APOE ε4 allele, and vitamin B12 levels were associated with incident anosmia over 5 years. Older age was associated with faster olfactory decline. Future studies with longer follow-ups are warranted.