Due to increasing changes in demographics, maintaining cognitive functioning later in life has become both economic and social concerns, and thus finding a cost-effective solution is one of the priorities in research. Factors like physical and intellectual activities have been associated with better cognitive performance in later life. While several studies have considered the impact of short-term physical activity interventions on cognitive functioning, retrospective research focusing on life-time physical activity experience has been sparse. The aim of the study was to determine the relationship between memory performance and whole brain matter integrity in seniors with different regular life-long physical activity experience. Fifty-three Latvian seniors aged 65–85 (M = 72.25, SD = 5.03, 83% female) with no self-reported chronic disease participated in the study. Measures of memory, physical activity and whole brain matter integrity were obtained and analysed. The obtained results indicated no significant relationship between physical activity experience and short and long-term memory and whole brain matter integrity; however, brain matter integrity was significantly correlated with demographic factors like age and education. These results might be related to inadequate physical activity measures, as well as unequal physical activity experience in participants. In the future, more detailed assessment of physical activity experience should be considered.