Obesity is associated with impaired memory in humans, and obesity induced by high-fat diets leads to cognitive deficits in rodents and in mouse models of Alzheimer’s disease (AD). However, it remains unclear how high-fat diets contribute to memory impairment. Therefore, we tested the effect of a high-fat diet on memory in male and female control non-transgenic (Non-Tg) and triple-transgenic AD (3xTgAD) mice and determined if a high-fat diet caused similar ultrastructural abnormalities to those observed in AD. Behavior was assessed in mice on control or high-fat diet at 4, 8, or 14 months of age and ultrastructural analysis at 8 months of age. A high-fat diet increased body weight, fat weight, and insulin levels with some differences in these metabolic responses observed between Non-Tg and 3xTgAD mice. In both sexes, high-fat feeding caused memory impairments in Non-Tg mice and accelerated memory deficits in 3xTgAD mice. In 3xTgAD mice, changes in hippocampal mitochondrial morphology were observed in capillaries and brain neuropil that were accompanied by a reduction in synapse number. A high-fat diet also caused mitochondria abnormalities and a reduction in synapse number in Non-Tg mice, but did not exacerbate the changes seen in 3xTgAD mice. Our data demonstrate that a high-fat diet affected memory in Non-Tg mice and produced similar impairments in mitochondrial morphology and synapse number comparable to those seen in AD mice, suggesting that the detrimental effects of a high-fat diet on memory might be due to changes in mitochondrial morphology leading to a reduction in synaptic number.