The average human brain is characterized by a global left-right asymmetry of shape, including a well-known 'torque' on the fronto-occipital axis, and less-studied differences on the dorsal-ventral axis. Torque has been claimed to be human-specific. However, the functional significance and developmental mechanisms underlying the global aspects of brain anatomical asymmetry are unknown. Here we used a registration-based approach with respect to a symmetrical template, to carry out the largest-ever analysis of global brain asymmetry in magnetic resonance imaging data. Three population datasets were used, the UK Biobank (N = 21,389), Human Connectome Project (N = 1,113) and BIL&GIN (N = 453). At the population level, there was an anterior and dorsal skew of the right hemisphere, relative to the left. Variances of this skew on the anterior-occipital and dorsal-ventral axes were largely independent, but both measures were associated with handedness, as well as various regional grey and white matter metrics. Both skew measures also showed heritabilities of 5%-10%, and four potential loci were identified in genome-wide association scanning. Several phenotypic variables related to early life experiences, cognitive functions, or mental health showed associations with the skew measures. These results provide replicable associations of global brain structural asymmetry measures with left-handedness, as well as insights into molecular genetic and early life factors which may contribute to brain asymmetry.