The current longitudinal study examined factors (sex, physical function, response to novelty, ability to adapt to a shift in light/dark cycle, brain connectivity), which might predict the emergence of impaired memory during aging. Male and female Fisher 344 rats were tested at 6, 12, and 18 months of age. Impaired spatial memory developed in middle-age (12 months), particularly in males, and the propensity for impairment increased with advanced age. A reduced response to novelty was observed over the course of aging, which is inconsistent with cross-sectional studies. This divergence likely resulted from differences in the history of environmental enrichment/impoverishment for cross-sectional and longitudinal studies. Animals that exhibited lower level exploration of the inner region on the open field test exhibited better memory at 12 months. Furthermore, males that exhibited a longer latency to enter a novel environment at 6 months, exhibited better memory at 12 months. For females, memory at 12 months was correlated with the ability to behaviorally adapt to a shift in light/dark cycle. Functional magnetic resonance imaging of the brain, conducted at 12 months, indicated that the decline in memory was associated with altered functional connectivity within different memory systems, most notably between the hippocampus and multiple regions such as the retrosplenial cortex, thalamus, striatum, and amygdala. Overall, some factors, specifically response to novelty at an early age and the capacity to adapt to shifts in light cycle, predicted spatial memory in middle-age, and spatial memory is associated with corresponding changes in brain connectivity. We discuss similarities and differences related to previous longitudinal and cross-sectional studies, as well as the role of sex differences in providing a theoretical framework to guide future longitudinal research on the trajectory of cognitive decline. In addition to demonstrating the power of longitudinal studies, these data highlight the importance of middle-age for identifying potential predictive indicators of sexual dimorphism in the trajectory in brain and cognitive aging.