Whilst many studies show sex differences in cerebral asymmetry, their mechanisms are still unknown. This report describes the potential impact of sex hormones and sex chromosomes by comparing MR data from 39 male and 47 female controls and 33 men with an extra X-chromosome (47,XXY).Methods: Regional asymmetry in gray and white matter volumes (GMV and WMV) was calculated using voxel based moprhometry (SPM5), by contrasting the unflipped and flipped individual GMV and WMV images. In addition, structural volumes were calculated for the thalamus, caudate, putamen, amygdala, and hippocampus, using the FreeSurfer software. Effects of plasma testosterone and estrogen on the GMV and WMV, as well on the right/left ratios of the subcortical volumes were tested by multi-regression analysis.Results: All three groups showed a leftward asymmetry in the motor cortex and the planum temporale, and a rightward asymmetry of the middle occipital cortex. Both asymmetries were more pronounced in 46,XY males than 46,XX females and 47,XXY males, and were positively correlated with testosterone levels. There was also a rightward asymmetry of the vermis and leftward GMV asymmetry in the cerebellar hemispheres in all groups. Notably, cerebellar asymmetries were larger in 46,XX females and 47,XXY males, but were not related to sex hormone levels. No asymmetry differences between 46,XX females and 47,XXY males, and no overall effects of brain size were detected.Conclusion: The asymmetry in the planum temporale area and the occipital cortex seem related to processes associated with testosterone, whereas the observed cerebellar asymmetries suggest a link with X-chromosome escapee genes. Sex differences in cerebral asymmetry are moderated by sex hormones and X-chromosome genes, in a regionally differentiated manner.