To investigate various risk factors of cognitive decline in the very old, we studied 494 subjects over 85 years old without diagnosis of dementia at baseline from the Tokyo Oldest Old Survey on Total Health, an ongoing, community-based cohort in Japan. Cognitive function was assessed at baseline and at 3-year followup using Mini-Mental State Examination (MMSE). Plasma samples were assayed for levels of cytomegalovirus (CMV) immunoglobulin G (IgG) antibodies, tumor necrosis factor-alpha, interleukin-6, and blood chemistry. Carotid artery plaques were measured using an ultrasonography. In the cross-sectional analyses using Tobit regression, individuals with high carotid artery plaque score (≥5.0) had MMSE scores that were 1.08 points lower compared to those with no plaque (95 % confidence interval (CI) −1.95 to −0.20; p = 0.016), adjusted for age, sex, and education. Individuals with CMV IgG titers in the highest quartile had MMSE scores that were 1.47 points lower compared to individuals in the lowest quartile (95 % CI −2.44 to −0.50; p = 0.003). CMV and carotid atherosclerosis showed evidence of an interaction, where the association between CMVand MMSE was present only in subjects with carotid artery plaque. In the longitudinal analyses using linear regression, carotid atherosclerosis, smoking, low grip strength, and poor activities of daily living (ADL) status were associated with faster cognitive decline, adjusted for age, sex, education, and baseline cognitive function. Our findings suggest that carotid atherosclerosis is consistently associated with low cognitive function in the very old and modifies the association between latent CMV infection and cognition.