Working memory relies critically on the dorsolateral prefrontal cortex (dlPFC). Morphology and function of the dlPFC, and corresponding working memory performance, are affected early in the aging process. However, these effects are heterogeneous, with nearly half of aged individuals spared of working memory deficits. Translationally relevant model systems are critical for investigating the neurobiological drivers of this variability and identifying why some people experience age-related working memory impairment while others do not. The common marmoset (Callithrix jacchus) is advantageous as a model in which to investigate the biological underpinnings of aging because, as a nonhuman primate, marmosets have a clearly defined dlPFC facilitating investigations of prefrontal-dependent cognitive functions, including working memory, and their short (~10 year) lifespan facilitates longitudinal studies of aging. Here, we conduct the first investigation of synaptic ultrastructure in the dlPFC of the marmoset and investigate whether there are changes to synaptic ultrastructure that are unique to aging with and without working memory impairment. To do this, we characterized working memory capacity in a cohort of marmosets that collectively covered their short lifespan, and found age-related working memory impairment. We also found a remarkable degree of heterogeneity in performance, similar to that found in humans. Utilizing three dimensional reconstruction from serial section electron microscopy, we visualized structural correlates of synaptic efficacy including boutons, mitochondria, and synapses in layer III of the dlPFC of three marmosets: one young adult (YA), one aged cognitively unimpaired (AU), and one aged cognitively impaired (AI). We find that aged marmosets have fewer synapses in dlPFC than young, and this is due to selective vulnerability of small synapses. Next, we tested the hypothesis that violation of the ultrastructural size principle underlies age-related working memory impairment. The ultrastructural size principle states that synaptic efficacy relies on coordinated scaling of synaptic components (e.g., synapses, mitochondria) with presynaptic boutons. While synapses and mitochondria scaled proportionally and were strongly correlated with presynaptic boutons in the YA and AU marmosets, the ultrastructural characteristics of the AI marmoset were alarmingly different. We found that age-related working memory impairment was associated with disproportionately large synapses compared to presynaptic boutons, specifically in those with mitochondria. Remarkably, presynaptic mitochondria and these boutons were completely decorrelated. We posit that this decorrelation results in mismatched energy supply and demand, leading to impaired synaptic transmission. This is the first report of age-related synapse loss in the marmoset, and the first demonstration that violation of the ultrastructural size principle underlies age-related working memory impairment.