Schizophrenia (SZ), major depressive disorder (MDD), and bipolar disorder (BD) are severe psychiatric disorders and share common characteristics not only in clinical symptoms but also in neuroimaging. The purpose of this study was to examine common and specific neuroanatomical features in individuals with these three psychiatric conditions. In this study, 70 patients with SZ, 85 patients with MDD, 42 patients with BD, and 95 healthy controls (HCs) were recruited. Voxel-based morphometry (VBM) analysis was used to explore brain imaging characteristics. Psychopathology was assessed using the Beck Depression Inventory (BDI), the Beck Anxiety Inventory (BAI), the Young Mania Rating Scale (YMRS), and the Positive and Negative Syndrome Scale (PANSS). Cognition was assessed using the digit symbol substitution test (DSST), forward-digital span (DS), backward-DS, and semantic fluency. Common reduced gray matter volume (GMV) in the orbitofrontal cortex (OFC) region was found across the SZ, MDD, and BD. Specific reduced GMV of brain regions was also found. For patients with SZ, we found reduced GMV in the frontal lobe, temporal pole, occipital lobe, thalamus, hippocampus, and cerebellum. For patients with MDD, we found reduced GMV in the frontal and temporal lobes, insular cortex, and occipital regions. Patients with BD had reduced GMV in the medial OFC, inferior temporal and fusiform regions, insular cortex, hippocampus, and cerebellum. Furthermore, the OFC GMV was correlated with processing speed as assessed with the DSST across four groups (r = 0.17, p = 0.004) and correlated with the PANSS positive symptoms sub-score in patients with SZ (r = − 0.27, p = 0.026). In conclusion, common OFC alterations in SZ, MDD, and BD provided evidence that this region dysregulation may play a critical role in the pathophysiology of these three psychiatric disorders.