In recent years, the functional importance of the anterior temporal lobes (ATLs) has come to prominence in two active, albeit unconnected branches of the literature. In one branch, neuropsychology and functional neuroimaging evidence emphasises the role of the ATLs in face recognition and linking faces to biographical knowledge. In the other, cognitive and clinical neuroscience investigations have shown that the ATLs are critical to all forms of semantic memory. To draw these literatures together and generate a unified account of ATL function, we test the predictions arising from each literature and examine the effects of bilateral versus unilateral ATL damage on face recognition, person knowledge and semantic memory. Sixteen people with bilateral ATL atrophy from semantic dementia (SD), 17 people with unilateral ATL resection for temporal lobe epilepsy (TLE; left=10, right=7), and 14 controls completed a test battery encompassing general semantic processing, person knowledge and perceptual face matching. SD patients were severely impaired across all semantic tasks, including person knowledge. Despite commensurate total ATL damage, unilateral resection generated mild impairments, with minimal differences between left- and right-ATL resection. Face matching performance was largely preserved but slightly reduced in SD and right TLE. All groups displayed the classic familiarity effect in face matching; however, this benefit was reduced in SD and right TLE groups and was aligned with the level of item-specific semantic knowledge in all participants. We propose a unified neurocognitive framework whereby the ATLs underpin a resilient bilateral representation system that supports semantic memory, person knowledge and face recognition.