Background: Relatively little is known regarding the association between objective measures of physical function such as cardiorespiratory fitness (CRF) and cognitive function tests in healthy older adults. Objective: To evaluate the relationship between CRF and cognitive function in adults aged 55 and older. Methods: Between 2008 and 2017, 4,931 men and women underwent a comprehensive preventive physical exam at the Cooper Clinic in Dallas, Texas. CRF was determined by duration of a maximal treadmill exercise test. Cognitive function was evaluated with the Montreal Cognitive Assessment (MoCA). In a multivariate model, adjusted odds ratios with 95% confidence intervals for MoCA scores < 26 (i.e., cognitive impairment) were determined by using CRF as both a continuous and a categorical variable. Results: The mean age of the sample was 61.0 ± 6.0 years; mean maximal MET values were 10.0 ± 2.2. Mean MoCA scores were 26.9 ± 2.2; 23.4% of the sample had MoCA scores indicative of cognitive impairment. The odds ratio for cognitive impairment was 0.93 (0.88–0.97) per 1-MET increment in CRF. When examined as a categorical variable, and using the lowest CRF quintile as the referent, there was a significantly reduced likelihood for cognitive impairment across the remaining ordered CRF categories (p trend = 0.004). Conclusion: The association between CRF and MoCA score in older adults suggests that meeting or exceeding public health guidelines for physical activity is likely to increase CRF in low fit individuals, maintain CRF in those with a moderate to high level of CRF, and thereby help to maintain cognitive function in healthy older adults.