Schizophrenia (SZ) and bipolar I disorder (BD) share common functional and structural brain abnormalities, as well as various degrees of social cognitive deficits, suggesting shared brain mechanisms. This study examined the relationship between social cognitive skills and structural brain integrity in 60 cases with SZ, 65 cases with psychotic BD, and 61 healthy controls (HCs). All participants underwent structural MRI and completed The Awareness of Social Inference Test (TASIT). Associations between social cognitive performance on each task of the TASIT (Emotion Recognition, Theory of Mind [ToM], and Complex ToM) and indices of gray matter volume and cortical thickness were investigated within three separate groups comprising (a) all cases with a history of psychosis (independently of their diagnostic status), (b) each diagnostic category separately, and (c) an HC group. Cases with psychosis showed worse social cognitive performance compared to the HC group, and SZ cases performed worse than BD on all ToM tasks, but not the Emotion Recognition subtest. Poor ToM performance was associated with thinner anterior temporal lobe in the combined group of cases with psychosis, and with decreased volume of the left fusiform gyrus/cerebellum in the HC group. In the BD group alone, poor ToM performance was associated with similar pattern of thinner anterior temporal lobe, as well as with increased volume in the mid-and posterior cingulate cortex/ precuneus. However, there were no associations between brain morphometry and social cognition in the SZ group when considered alone.Taken together, ToM behavioral deficits were associated with thinner right anterior temporal lobe, a critical region from a larger affective ToM network among all cases with psychosis, but aberrant morphology of brain regions key for self-processing were specific to the BD group. These findings may have important implications for targeted interventions for the affective social brain network in BD.