Several studies report that caloric restriction (CR) or intermittent fasting (IF) can improve cognition, while others report limited or no cognitive benefits. Here, we compare the effects of 20% CR, 40% CR, 1-day IF, and 2-day IF feeding paradigms to ad libitum controls (AL) on Y-maze working memory and contextual fear memory (CFM) in a large population of Diversity Outbred mice that model the genetic diversity of humans. While CR and IF interventions improve lifespan, we observed no enhancement of working memory or CFM in mice on these feeding paradigms, and report 40% CR to be damaging in the context of long-term memory. Using Quantitative Trait Loci mapping, we identified the gene Slc16a7 to be associated with late-life long-term memory outcomes in mice on lifespan promoting feeding paradigms. Limited utility of dieting and fasting on memory in mice that recapitulate genetic diversity in the human population highlights the need for anti-aging therapeutics that promote cognitive function, with a neuronal monocarboxylate transporter encoded by Slc16a7 highlighted as novel target.