The ageing process is associated with reduced emotional recognition (ER) performance. The ER ability is an essential part of non-verbal communication, and its role is crucial for proper social functioning. Here, using the ‘Cambridge Centre for Ageing and Neuroscience cohort sample’, we investigated when ER, measured using a facial emotion recognition test, begins to consistently decrease along the lifespan. Moreover, using structural and functional MRI data, we identified the neural correlates associated with ER maintenance in the age groups showing early signs of ER decline (N = 283; age range: 58–89 years). The ER performance was positively correlated with greater volume in the superior parietal lobule, higher white matter integrity in the corpus callosum and greater functional connectivity in the mid-cingulate area. Our results suggest that higher ER accuracy in older people is associated with preserved gray and white matter volumes in cognitive or interconnecting areas, subserving brain regions directly involved in emotional processing.