Decline in spatial context memory emerges in midlife, the time when most females transition from pre- to post-menopause. Recent evidence suggests that, among post-menopausal females, advanced age is associated with functional brain alterations and lower spatial context memory. However, it is unknown whether similar effects are evident for white matter (WM) and, moreover, whether such effects contribute to sex differences at midlife. To address this, we conducted a study on 96 cognitively unimpaired middle-aged adults (30 males, 32 pre-menopausal females, 34 post-menopausal females). Spatial context memory was assessed using a face-location memory paradigm, while WM microstructure was assessed using diffusion tensor imaging. Behaviorally, advanced age was associated with lower spatial context memory in post- menopausal females but not pre-menopausal females or males. Additionally, advanced age was associated with microstructural variability in predominantly frontal WM (e.g., anterior corona radiata, genu of corpus callosum), which was related to lower spatial context memory among post-menopausal females. Our findings suggest that post-menopausal status enhances vulnerability to age effects on the brain's WM and episodic memory.