Background and Objectives: Substance use disorder (SUD) includes maladaptive patterns of substance use despite negative consequences. Previous structural neuroimaging studies showed some structural alterations in SUD, but it remains unknown whether these alterations are specifically associated with SUD or common comorbidities. This study attempts to validate the findings of structural differences between SUD, healthy controls (HC), and psychiatric controls (PC). Methods: We used HC (N = 86) matched for demographics, and PC (N = 86) matched for demographics and psychiatric diagnoses to a group of SUD patients (N = 86). We assessed the group differences of subcortical volumes, cortical volumes, thickness, and surface areas between SUD and HC. We then analyzed the group differences between SUD and PC within regions showing differences between SUD and HC. Results: SUD had smaller left nucleus accumbens, right thalamus, right hippocampus, left caudal anterior cingulate cortex (ACC) volume, and larger right caudal ACC volume, and right caudal ACC, right caudal middle frontal gyrus (MFG), and right posterior cingulate cortex (PCC) surface than HC. Increased right caudal ACC volume and right PCC surface in SUD were the only findings when compared with PC. Several areas showed thickness alterations between SUD and HC, but none survived multiple comparisons vs PC. Discussion and Conclusions: Our findings suggest that cingulate structures may be altered in SUD compared with both HC and PC. Scientific Significance: These results are among the first to indicate that some structural alterations may be SUD-specific, and highlight a cautionary note about using HC in psychiatric biomarker research.