Comparative study of the structural asymmetry of the human and chimpanzee brain may shed light on the evolution of language and other cognitive abilities in humans. Here we report the results of vertex-wise and ROI-based analyses that compared surface area (SA) and cortical thickness (CT) asymmetries in 3D MR images obtained for 91 humans and 77 chimpanzees. The human brain is substantially more asymmetric than the chimpanzee brain. In particular, the human brain has 1) larger total SA in the right compared with the left cerebral hemisphere, 2) a global torque-like asymmetry pattern of widespread thicker cortex in the left compared with the right frontal and the right compared with the left temporo-parieto-occipital lobe, and 3) local asymmetries, most notably in medial occipital cortex and superior temporal gyrus, where rightward asymmetry is observed for both SA and CT. There is also 4) a prominent asymmetry specific to the chimpanzee brain, namely, rightward CT asymmetry of precentral cortex. These findings provide evidence of there being substantial differences in asymmetry between the human and chimpanzee brain. The unique asymmetries of the human brain are potential neural substrates for cognitive specializations, and the presence of significant CT asymmetry of precentral gyrus in the chimpanzee brain should be further investigated.