Inferring the intentions and emotions of other people from their observed behavior is a critical component of social cognition. While neuroimaging studies have identified a distributed set of brain regions involved in social inference, it remains unknown whether performing social inference is an abstract computation that generalizes across different stimulus categories, or is instead entangled with specific stimuli such as faces. While social inference processes are thought to be distinct from those supporting other executive functions, their stimulus-domain specificity has not been tested. We recorded single-neuron activity from the medial temporal lobe (MTL) and the medial frontal cortex (MFC) in neurosurgical patients. Subjects switched between making different types of social inferences from images of faces, hands, and natural scenes. We found largely distinct populations of neurons that encoded inference type for social (faces, hands) and nonsocial (scenes) stimuli in both brain regions. In contrast, stimulus category was itself represented in a task-general manner in both MTL and MFC. Uniquely in the MTL, social inference type was represented by separate subsets of neurons for faces and hands, resulting in a domain-specific representation. These results reveal evidence for specialized social inference processes in the MTL, in which inference representations were entangled with stimulus type as expected from a domain-specific process.