Older adults (OA), relative to young adults (YA), exhibit age-related alterations in functional Magnetic Resonance Imaging (fMRI) activity during associative encoding, which contributes to deficits in source memory. Yet, there are remarkable individual differences in brain health and memory performance among OA. Cardiorespiratory fitness (CRF) is one individual difference factor that may attenuate brain aging, and thereby contribute to enhanced source memory in OA. To examine this possibility, 26 OA and 31 YA completed a treadmill-based exercise test to evaluate CRF (peak VO2) and fMRI to examine brain activation during a face-name associative encoding task. Our results indicated that in OA, peak VO2 was positively associated with fMRI activity during associative encoding in multiple regions including bilateral prefrontal cortex, medial frontal cortex, bilateral thalamus and left hippocampus. Next, a conjunction analysis was conducted to assess whether CRF influenced age-related differences in fMRI activation. We classified OA as high or low CRF and compared their activation to YA. High fit OA (HFOA) showed fMRI activation more similar to YA than low fit OA (LFOA) (i.e., reduced age-related differences) in multiple regions including thalamus, posterior and prefrontal cortex. Conversely, in other regions, primarily in prefrontal cortex, HFOA, but not LFOA, demonstrated greater activation than YA (i.e., increased age-related differences). Further, fMRI activity in these brain regions was positively associated with source memory among OA, with a mediation model demonstrating that associative encoding activation in medial frontal cortex indirectly influenced the relationship between peak VO2 and subsequent source memory performance. These results indicate that CRF may contribute to neuroplasticity among OA, reducing age-related differences in some brain regions, consistent with the brain maintenance hypothesis, but accentuating age-differences in other regions, consistent with the brain compensation hypothesis.