Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by behavioral impairment in social interactions. Although theoretical and empirical evidence suggests that impairment in the social brain network could be the neural underpinnings of ASD, previous structural magnetic resonance imaging (MRI) studies in adults with ASD have not provided clear support for this, possibly due to confounding factors, such as language impairments. To further explore this issue, we acquired structural MRI data and analyzed gray matter volume in adults with ASD (n = 36) who had no language impairments (diagnosed with Asperger’s disorder or pervasive developmental disorder not otherwise specified, with symptoms milder than those of Asperger’s disorder), had no comorbidity, and were not taking medications, and in age- and sex-matched typically developing (TD) controls (n = 36). Univariate voxel-based morphometry analyses revealed that regional gray matter volume was lower in the ASD than in the control group in several brain regions, including the right inferior occipital gyrus, left fusiform gyrus, right middle temporal gyrus, bilateral amygdala, right inferior frontal gyrus, right orbitofrontal cortex, and left dorsomedial prefrontal cortex. A multivariate approach using a partial least squares (PLS) method showed that these regions constituted a network that could be used to discriminate between the ASD and TD groups. A PLS discriminant analysis using information from these regions showed high accuracy, sensitivity, specificity, and precision (>80%) in discriminating between the groups. These results suggest that reduced gray matter volume in the social brain network represents the neural underpinnings of behavioral social malfunctioning in adults with ASD.